2023-09-29T22:50:42.249 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-29T22:50:42.249 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --description rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/dashboard} --verbose --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517 -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/orig.config.yaml 2023-09-29T22:50:42.288 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T22:50:42.362 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/dashboard} email: yweinste@redhat.com first_in_suite: false job_id: '7406517' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug client: 20 debug mgrc: 20 debug ms: 1 global: mon election default strategy: 1 mds: debug mgrc: 20 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug mgrc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mgr.x - mon.a - mon.c - mds.a - mds.c - osd.0 - client.0 - - mgr.y - mgr.z - mon.b - mds.b - osd.1 - osd.2 - osd.3 - client.1 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4/SG/jST+P3fpYnvP3qqBOCdxntMhW67oJi2kYDwucLwJ6BPd99WBtmrWt5kYaiaQ0UR/qpZ7k/fy4wK9GW/g= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArobUAH9Zdgerb63Cfvmh72MtB0kienk5a9ay4DNdK59tgLV3YNl1ZfokzHBBrrcpYP12D7NyCVsOU0Gco9sTA= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - sudo sed -i 's/runtime = "runc"/#runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/#runtime = "crun"/runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - ceph: log-ignorelist: - overall HEALTH_ - \(MGR_DOWN\) - \(PG_ - replacing it with standby - No standby daemons available - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_DOWN\) - \(OSD_HOST_DOWN\) - \(POOL_APP_NOT_ENABLED\) - \(OSDMAP_FLAGS\) - \(OSD_FLAGS\) - \(TELEMETRY_CHANGED\) - pauserd,pausewr flag\(s\) set - Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running - evicting unresponsive client .+ - MON_DOWN wait-for-scrub: false - rgw: - client.0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.mgr.test_dashboard - tasks.mgr.dashboard.test_api - tasks.mgr.dashboard.test_auth - tasks.mgr.dashboard.test_cephfs - tasks.mgr.dashboard.test_cluster - tasks.mgr.dashboard.test_cluster_configuration - tasks.mgr.dashboard.test_crush_rule - tasks.mgr.dashboard.test_erasure_code_profile - tasks.mgr.dashboard.test_health - tasks.mgr.dashboard.test_host - tasks.mgr.dashboard.test_logs - tasks.mgr.dashboard.test_mgr_module - tasks.mgr.dashboard.test_monitor - tasks.mgr.dashboard.test_motd - tasks.mgr.dashboard.test_orchestrator - tasks.mgr.dashboard.test_osd - tasks.mgr.dashboard.test_perf_counters - tasks.mgr.dashboard.test_pool - tasks.mgr.dashboard.test_rbd - tasks.mgr.dashboard.test_rbd_mirroring - tasks.mgr.dashboard.test_requests - tasks.mgr.dashboard.test_rgw - tasks.mgr.dashboard.test_role - tasks.mgr.dashboard.test_settings - tasks.mgr.dashboard.test_summary - tasks.mgr.dashboard.test_telemetry - tasks.mgr.dashboard.test_user teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-29T22:50:42.362 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-29T22:50:42.363 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-29T22:50:42.363 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-29T22:50:42.365 INFO:teuthology.task.internal:Checking packages... 2023-09-29T22:50:42.386 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-29T22:50:42.386 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-29T22:50:42.386 INFO:teuthology.packaging:ref: None 2023-09-29T22:50:42.386 INFO:teuthology.packaging:tag: None 2023-09-29T22:50:42.387 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T22:50:42.387 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T22:50:42.387 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T22:50:42.617 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-29T22:50:42.617 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-29T22:50:42.626 INFO:teuthology.task.internal:no buildpackages task found 2023-09-29T22:50:42.626 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-29T22:50:42.680 INFO:teuthology.task.internal:Saving configuration 2023-09-29T22:50:42.695 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-29T22:50:42.706 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T22:50:42.732 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:45:37.690018', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4/SG/jST+P3fpYnvP3qqBOCdxntMhW67oJi2kYDwucLwJ6BPd99WBtmrWt5kYaiaQ0UR/qpZ7k/fy4wK9GW/g='} 2023-09-29T22:50:42.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:45:37.691951', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArobUAH9Zdgerb63Cfvmh72MtB0kienk5a9ay4DNdK59tgLV3YNl1ZfokzHBBrrcpYP12D7NyCVsOU0Gco9sTA='} 2023-09-29T22:50:42.752 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-29T22:50:42.763 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-09-29T22:50:42.763 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-09-29T22:50:42.763 INFO:teuthology.run_tasks:Running task console_log... 2023-09-29T22:50:42.841 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9f02dbfb80>, signals=[15]) 2023-09-29T22:50:42.841 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-29T22:50:42.852 INFO:teuthology.task.internal:Opening connections... 2023-09-29T22:50:42.852 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2023-09-29T22:50:42.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:50:42.924 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T22:50:42.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:50:42.991 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-29T22:50:43.005 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-09-29T22:50:43.027 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-09-29T22:50:43.027 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-09-29T22:50:43.088 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2023-09-29T22:50:43.088 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-29T22:50:43.089 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-29T22:50:43.090 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-09-29T22:50:43.114 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2023-09-29T22:50:43.139 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2023-09-29T22:50:43.139 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2023-09-29T22:50:43.200 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2023-09-29T22:50:43.200 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-29T22:50:43.201 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-29T22:50:43.202 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2023-09-29T22:50:43.222 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-29T22:50:43.236 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-29T22:50:43.253 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-29T22:50:43.253 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2023-09-29T22:50:43.256 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2023-09-29T22:50:43.277 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-29T22:50:43.287 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-29T22:50:43.287 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2023-09-29T22:50:43.317 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2023-09-29T22:50:43.352 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-29T22:50:43.405 INFO:teuthology.run_tasks:Running task kernel... 2023-09-29T22:50:43.428 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-29T22:50:43.428 INFO:teuthology.task.kernel:config {'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-29T22:50:43.428 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T22:50:43.429 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T22:50:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:50:43.448 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2023-09-29T22:50:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:50:43.450 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-09-29T22:50:43.512 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-489.el8.x86_64 2023-09-29T22:50:43.512 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.18.0-489.el8.x86_64 2023-09-29T22:50:43.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2023-09-29T22:50:43.513 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-489.el8.x86_64 2023-09-29T22:50:43.513 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.18.0-489.el8.x86_64 2023-09-29T22:50:43.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2023-09-29T22:50:48.094 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 17 MB/s | 33 MB 00:01 2023-09-29T22:50:48.374 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 15 MB/s | 33 MB 00:02 2023-09-29T22:51:06.192 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 4.0 MB/s | 48 MB 00:11 2023-09-29T22:51:06.347 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 4.0 MB/s | 48 MB 00:11 2023-09-29T22:51:13.633 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2023-09-29T22:51:13.738 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2023-09-29T22:51:14.193 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-09-29T22:51:14.280 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-09-29T22:51:14.553 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2023-09-29T22:51:14.565 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-09-29T22:51:15.968 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-09-29T22:51:16.987 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 16 MB 00:02 2023-09-29T22:51:19.799 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 350 kB/s | 24 kB 00:00 2023-09-29T22:51:20.829 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 243 kB/s | 24 kB 00:00 2023-09-29T22:51:23.289 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-29T22:51:23.289 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-29T22:51:23.560 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T22:51:23.560 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-29T22:51:23.561 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T22:51:23.562 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T22:51:23.562 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T22:51:23.562 INFO:teuthology.orchestra.run.smithi121.stdout:Install 3 Packages 2023-09-29T22:51:23.562 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T22:51:23.562 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 88 M 2023-09-29T22:51:23.563 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 96 M 2023-09-29T22:51:23.563 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T22:51:24.436 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-29T22:51:24.436 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-29T22:51:24.691 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-29T22:51:24.692 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-29T22:51:24.693 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T22:51:24.693 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T22:51:24.693 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T22:51:24.693 INFO:teuthology.orchestra.run.smithi123.stdout:Install 3 Packages 2023-09-29T22:51:24.693 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T22:51:24.693 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 88 M 2023-09-29T22:51:24.694 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 96 M 2023-09-29T22:51:24.694 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T22:51:28.466 INFO:teuthology.orchestra.run.smithi121.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 2.1 MB/s | 10 MB 00:04 2023-09-29T22:51:29.446 INFO:teuthology.orchestra.run.smithi123.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 2.2 MB/s | 10 MB 00:04 2023-09-29T22:51:39.862 INFO:teuthology.orchestra.run.smithi123.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 2.4 MB/s | 35 MB 00:14 2023-09-29T22:51:40.880 INFO:teuthology.orchestra.run.smithi121.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 2.1 MB/s | 35 MB 00:17 2023-09-29T22:51:41.639 INFO:teuthology.orchestra.run.smithi121.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 2.4 MB/s | 43 MB 00:17 2023-09-29T22:51:41.639 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:51:41.639 INFO:teuthology.orchestra.run.smithi121.stdout:Total 4.9 MB/s | 88 MB 00:18 2023-09-29T22:51:42.308 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T22:51:42.759 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T22:51:42.759 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T22:51:43.528 INFO:teuthology.orchestra.run.smithi123.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 2.3 MB/s | 43 MB 00:18 2023-09-29T22:51:43.528 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T22:51:43.529 INFO:teuthology.orchestra.run.smithi123.stdout:Total 4.7 MB/s | 88 MB 00:18 2023-09-29T22:51:43.990 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T22:51:44.019 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T22:51:44.194 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T22:51:44.617 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T22:51:44.617 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T22:51:45.852 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T22:51:45.890 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T22:51:46.808 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T22:51:47.398 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T22:51:48.749 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T22:51:49.283 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T22:51:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T22:51:51.372 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T22:51:52.937 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T22:51:53.366 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T22:51:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T22:51:55.068 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:51:56.747 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T22:51:57.028 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:16.862 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:16.886 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-29T22:52:17.395 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-29T22:52:18.092 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:18.092 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T22:52:18.092 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T22:52:18.093 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-29T22:52:18.093 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-29T22:52:18.093 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T22:52:18.093 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T22:52:18.249 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:52:18.464 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T22:52:18.464 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T22:52:18.464 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-29T22:52:18.465 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T22:52:18.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T22:52:18.465 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-09-29T22:52:18.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T22:52:18.465 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:52:18.671 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T22:52:18.671 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T22:52:18.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2023-09-29T22:52:18.704 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2023-09-29T22:52:18.704 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.02 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 129.el8 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Group : System Environment/Base 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 9265586 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-29T22:52:18.705 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:Relocations : (not relocatable) 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : CentOS BuildSystem 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T22:52:18.706 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T22:52:18.707 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T22:52:18.707 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2023-09-29T22:52:18.707 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T22:52:18.707 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T22:52:18.708 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T22:52:18.708 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 4.18.0-514.el8.x86_64 2023-09-29T22:52:18.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T22:52:18.869 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:18.894 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:19.385 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:19.385 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-29T22:52:19.385 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-29T22:52:20.119 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T22:52:20.119 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T22:52:20.119 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T22:52:20.119 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-29T22:52:20.119 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-29T22:52:20.119 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T22:52:20.120 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T22:52:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2023-09-29T22:52:20.258 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:52:20.474 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T22:52:20.474 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T22:52:20.475 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-29T22:52:20.475 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T22:52:20.475 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T22:52:20.475 INFO:teuthology.task.kernel:Installing distro kernel on mgr.y... 2023-09-29T22:52:20.475 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T22:52:20.475 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T22:52:20.681 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T22:52:20.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T22:52:20.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2023-09-29T22:52:20.714 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2023-09-29T22:52:20.714 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2023-09-29T22:52:20.714 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.02 2023-09-29T22:52:20.714 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 129.el8 2023-09-29T22:52:20.714 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Group : System Environment/Base 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 9265586 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Relocations : (not relocatable) 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : CentOS BuildSystem 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2023-09-29T22:52:20.715 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T22:52:20.716 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T22:52:20.717 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T22:52:20.717 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 4.18.0-514.el8.x86_64 2023-09-29T22:52:20.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T22:52:22.023 INFO:teuthology.orchestra.run.smithi121.stderr:done 2023-09-29T22:52:22.024 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-09-29T22:52:22.047 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.ZjO9MzKXdL 2023-09-29T22:52:22.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZjO9MzKXdL 2023-09-29T22:52:22.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.ZjO9MzKXdL 2023-09-29T22:52:22.148 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2023-09-29T22:52:22.272 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.ZjO9MzKXdL is 6KB 2023-09-29T22:52:22.320 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.ZjO9MzKXdL 2023-09-29T22:52:22.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T22:52:22.402 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-29T22:52:22.403 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-29T22:52:22.403 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-29T22:52:22.403 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-29T22:52:22.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-29T22:52:22.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2023-09-29T22:52:24.138 INFO:teuthology.orchestra.run.smithi123.stderr:done 2023-09-29T22:52:24.140 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-09-29T22:52:24.173 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.E4xIUUl8lg 2023-09-29T22:52:24.174 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.E4xIUUl8lg 2023-09-29T22:52:24.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.E4xIUUl8lg 2023-09-29T22:52:24.358 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.E4xIUUl8lg is 6KB 2023-09-29T22:52:24.406 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.E4xIUUl8lg 2023-09-29T22:52:24.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T22:52:24.498 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-29T22:52:24.499 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-29T22:52:24.499 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-29T22:52:24.499 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-29T22:52:24.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-29T22:52:24.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2023-09-29T22:52:52.582 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-09-29T22:52:52.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-29T22:52:52.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:52:54.674 INFO:teuthology.task.kernel:Checking client mgr.y for new kernel version... 2023-09-29T22:52:54.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-09-29T22:52:54.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:53:10.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-09-29T22:53:13.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-09-29T22:53:19.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-29T22:53:19.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:53:22.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-09-29T22:53:22.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:53:22.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-09-29T22:53:25.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-09-29T22:53:34.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-29T22:53:34.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:53:37.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-09-29T22:53:37.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:53:53.402 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-09-29T22:53:55.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-09-29T22:54:08.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-29T22:54:08.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:54:09.831 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-09-29T22:54:10.384 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-29T22:54:10.384 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "4.18.0-514.el8.x86_64"... 2023-09-29T22:54:10.384 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2023-09-29T22:54:10.402 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-514.el8.x86_64 2023-09-29T22:54:10.402 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-29T22:54:10.402 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T22:54:10.402 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-29T22:54:10.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-09-29T22:54:10.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T22:54:11.403 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-09-29T22:54:11.403 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T22:54:11.472 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2023-09-29T22:54:11.489 DEBUG:teuthology.parallel:result is None 2023-09-29T22:54:11.804 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-09-29T22:54:12.377 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-09-29T22:54:12.377 INFO:teuthology.task.kernel:Checking kernel version of mgr.y, want "4.18.0-514.el8.x86_64"... 2023-09-29T22:54:12.377 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-09-29T22:54:12.394 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-514.el8.x86_64 2023-09-29T22:54:12.394 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-29T22:54:12.394 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T22:54:12.394 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-29T22:54:13.395 INFO:teuthology.task.kernel:Enabling kdb on mgr.y... 2023-09-29T22:54:13.395 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T22:54:13.474 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2023-09-29T22:54:13.488 DEBUG:teuthology.parallel:result is None 2023-09-29T22:54:13.488 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-29T22:54:13.499 INFO:teuthology.task.internal:Creating test directory... 2023-09-29T22:54:13.500 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T22:54:13.502 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T22:54:13.522 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-29T22:54:13.530 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-29T22:54:13.545 INFO:teuthology.task.internal:Creating archive directory... 2023-09-29T22:54:13.546 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T22:54:13.565 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T22:54:13.597 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-29T22:54:13.603 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-29T22:54:13.604 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T22:54:13.633 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T22:54:13.672 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T22:54:13.673 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T22:54:13.683 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T22:54:13.684 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T22:54:13.685 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-29T22:54:13.693 INFO:teuthology.task.internal:Configuring sudo... 2023-09-29T22:54:13.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T22:54:13.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T22:54:13.757 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-29T22:54:13.765 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-29T22:54:13.765 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T22:54:13.800 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T22:54:13.818 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T22:54:13.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T22:54:13.956 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T22:54:14.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T22:54:14.084 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T22:54:14.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T22:54:14.151 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T22:54:14.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T22:54:14.251 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T22:54:14.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T22:54:14.379 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T22:54:14.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T22:54:14.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2023-09-29T22:54:14.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2023-09-29T22:54:14.497 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T22:54:14.560 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T22:54:15.022 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-29T22:54:15.033 INFO:teuthology.task.internal:Starting timer... 2023-09-29T22:54:15.033 INFO:teuthology.run_tasks:Running task pcp... 2023-09-29T22:54:15.069 INFO:teuthology.run_tasks:Running task selinux... 2023-09-29T22:54:15.096 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-29T22:54:15.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2023-09-29T22:54:15.170 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2023-09-29T22:54:15.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2023-09-29T22:54:15.236 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2023-09-29T22:54:15.238 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-29T22:54:15.241 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2023-09-29T22:54:15.261 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2023-09-29T22:54:15.262 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2023-09-29T22:54:15.283 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2023-09-29T22:54:15.283 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive'} 2023-09-29T22:54:15.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T22:54:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:54:15.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T22:54:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T22:54:15.356 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-29T22:54:15.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2023-09-29T22:54:15.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2023-09-29T22:54:15.424 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-29T22:54:15.438 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-29T22:54:15.444 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-29T22:54:15.573 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-29T22:54:15.588 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-29T22:54:15.590 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-29T23:01:37.923 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2023-09-29T23:01:37.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-29T23:01:37.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:01:37.998 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-09-29T23:01:38.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-29T23:01:38.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-09-29T23:01:38.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:01:38.134 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-09-29T23:01:38.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-09-29T23:01:38.206 INFO:teuthology.run_tasks:Running task clock... 2023-09-29T23:01:38.216 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-29T23:01:38.217 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T23:01:38.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:01:38.220 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T23:01:38.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:01:38.250 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T23:01:38.264 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T23:01:38.279 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T23:01:38.294 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T23:01:38.324 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2023-09-29T23:01:38.335 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2023-09-29T23:01:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T23:01:38.356 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2023-09-29T23:01:38.362 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T23:01:38.367 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2023-09-29T23:01:38.380 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T23:01:38.394 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T23:01:38.427 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-09-29T23:01:38.430 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:01:38.430 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-09-29T23:01:38.430 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.430 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.430 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.430 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.459 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-09-29T23:01:38.462 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:01:38.462 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-09-29T23:01:38.462 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.462 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.462 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.463 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:01:38.464 INFO:teuthology.run_tasks:Running task pexec... 2023-09-29T23:01:38.473 INFO:teuthology.task.pexec:Executing custom commands... 2023-09-29T23:01:38.474 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi121.front.sepia.ceph.com 2023-09-29T23:01:38.474 DEBUG:teuthology.orchestra.run.smithi121:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-09-29T23:01:38.474 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:01:38.474 DEBUG:teuthology.orchestra.run.smithi123:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-09-29T23:01:39.352 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Sep 2023 10:59:55 PM UTC. 2023-09-29T23:01:39.377 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Sep 2023 10:59:55 PM UTC. 2023-09-29T23:01:41.371 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:01:41.371 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout:Resetting modules: 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:01:41.372 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:01:41.373 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:01:41.373 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-09-29T23:01:41.373 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:01:41.373 INFO:teuthology.orchestra.run.smithi123.stdout:Resetting modules: 2023-09-29T23:01:41.374 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools 2023-09-29T23:01:41.374 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:01:41.374 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:01:41.374 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:01:41.374 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:01:41.616 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:01:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:01:42.556 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Sep 2023 10:59:55 PM UTC. 2023-09-29T23:01:42.575 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Sep 2023 10:59:55 PM UTC. 2023-09-29T23:01:44.540 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:01:44.562 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:01:44.697 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout:Installing group/module packages: 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-09-29T23:01:44.698 INFO:teuthology.orchestra.run.smithi121.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-09-29T23:01:44.699 INFO:teuthology.orchestra.run.smithi121.stdout:Installing module profiles: 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools/common 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools rhel8 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2023-09-29T23:01:44.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:01:44.701 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 26 M 2023-09-29T23:01:44.701 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 87 M 2023-09-29T23:01:44.701 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout:Installing group/module packages: 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-09-29T23:01:44.719 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-09-29T23:01:44.720 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2023-09-29T23:01:44.721 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-09-29T23:01:44.721 INFO:teuthology.orchestra.run.smithi123.stdout:Installing module profiles: 2023-09-29T23:01:44.721 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools/common 2023-09-29T23:01:44.721 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:01:44.721 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling module streams: 2023-09-29T23:01:44.721 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools rhel8 2023-09-29T23:01:44.721 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:01:44.722 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:01:44.722 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:01:44.722 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2023-09-29T23:01:44.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:01:44.722 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 26 M 2023-09-29T23:01:44.723 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 87 M 2023-09-29T23:01:44.723 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:01:45.369 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 1.5 MB/s | 739 kB 00:00 2023-09-29T23:01:45.391 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 1.5 MB/s | 739 kB 00:00 2023-09-29T23:01:45.410 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 1.9 MB/s | 1.0 MB 00:00 2023-09-29T23:01:45.424 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 1.9 MB/s | 1.0 MB 00:00 2023-09-29T23:01:45.452 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 454 kB/s | 38 kB 00:00 2023-09-29T23:01:45.466 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 504 kB/s | 38 kB 00:00 2023-09-29T23:01:45.503 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 2.5 MB/s | 239 kB 00:00 2023-09-29T23:01:45.508 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 2.8 MB/s | 239 kB 00:00 2023-09-29T23:01:45.595 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-podman-4.6.0-1.module_el8+582+a 1.9 MB/s | 177 kB 00:00 2023-09-29T23:01:45.600 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-podman-4.6.0-1.module_el8+582+a 1.9 MB/s | 177 kB 00:00 2023-09-29T23:01:45.701 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 248 kB/s | 25 kB 00:00 2023-09-29T23:01:45.753 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): buildah-1.31.3-1.module_el8+664+4072b3a 10 MB/s | 8.8 MB 00:00 2023-09-29T23:01:45.770 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 141 kB/s | 25 kB 00:00 2023-09-29T23:01:45.809 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): buildah-1.31.3-1.module_el8+664+4072b3a 9.7 MB/s | 8.8 MB 00:00 2023-09-29T23:01:45.834 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 703 kB/s | 94 kB 00:00 2023-09-29T23:01:45.837 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): netavark-1.7.0-1.module_el8+487+8e42a27 9.7 MB/s | 3.7 MB 00:00 2023-09-29T23:01:45.863 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 857 kB/s | 94 kB 00:00 2023-09-29T23:01:45.893 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): netavark-1.7.0-1.module_el8+487+8e42a27 8.8 MB/s | 3.7 MB 00:00 2023-09-29T23:01:45.938 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): udica-0.2.6-20.module_el8+487+8e42a277 704 kB/s | 53 kB 00:00 2023-09-29T23:01:45.987 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): udica-0.2.6-20.module_el8+487+8e42a277 560 kB/s | 53 kB 00:00 2023-09-29T23:01:46.013 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 545 kB/s | 41 kB 00:00 2023-09-29T23:01:46.080 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): toolbox-0.0.99.4-5.module_el8+649+e2de 10 MB/s | 2.5 MB 00:00 2023-09-29T23:01:46.087 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): toolbox-0.0.99.4-5.module_el8+649+e2de 10 MB/s | 2.5 MB 00:00 2023-09-29T23:01:46.133 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 282 kB/s | 41 kB 00:00 2023-09-29T23:01:46.239 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): skopeo-1.13.3-1.module_el8+664+4072b3a 17 MB/s | 8.1 MB 00:00 2023-09-29T23:01:46.239 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:01:46.239 INFO:teuthology.orchestra.run.smithi121.stdout:Total 17 MB/s | 26 MB 00:01 2023-09-29T23:01:46.288 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): skopeo-1.13.3-1.module_el8+664+4072b3a 17 MB/s | 8.1 MB 00:00 2023-09-29T23:01:46.288 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:01:46.289 INFO:teuthology.orchestra.run.smithi123.stdout:Total 16 MB/s | 26 MB 00:01 2023-09-29T23:01:46.504 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:01:46.525 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:01:46.525 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:01:46.551 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:01:46.572 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:01:46.572 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:01:46.773 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:01:46.776 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:01:46.816 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:01:46.819 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:01:47.428 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:01:47.512 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:01:47.682 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-09-29T23:01:47.725 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-09-29T23:01:47.853 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-09-29T23:01:47.894 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-09-29T23:01:48.049 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-09-29T23:01:48.072 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-09-29T23:01:48.280 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-09-29T23:01:48.304 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-09-29T23:01:48.838 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-09-29T23:01:48.846 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-09-29T23:01:49.733 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-09-29T23:01:49.788 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-09-29T23:01:49.961 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-09-29T23:01:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-09-29T23:01:50.172 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-09-29T23:01:50.264 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-09-29T23:01:50.432 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-09-29T23:01:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-09-29T23:01:50.845 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-09-29T23:01:50.952 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-09-29T23:01:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-09-29T23:01:51.838 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-09-29T23:01:51.947 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-09-29T23:01:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-09-29T23:01:52.134 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-09-29T23:01:52.276 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-09-29T23:01:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-09-29T23:01:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-09-29T23:01:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-09-29T23:01:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-09-29T23:01:52.697 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-09-29T23:01:52.698 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-09-29T23:01:52.699 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-09-29T23:01:52.699 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-09-29T23:01:53.449 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-09-29T23:01:53.449 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:01:53.449 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-09-29T23:01:53.450 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-09-29T23:01:53.451 INFO:teuthology.orchestra.run.smithi121.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-09-29T23:01:53.451 INFO:teuthology.orchestra.run.smithi121.stdout: yajl-2.1.0-12.el8.x86_64 2023-09-29T23:01:53.451 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:01:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:01:53.760 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-09-29T23:01:53.760 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:01:53.760 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:01:53.760 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-09-29T23:01:53.760 INFO:teuthology.orchestra.run.smithi123.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-09-29T23:01:53.760 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-09-29T23:01:53.760 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: yajl-2.1.0-12.el8.x86_64 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:01:53.761 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:01:53.771 DEBUG:teuthology.parallel:result is None 2023-09-29T23:01:54.146 DEBUG:teuthology.parallel:result is None 2023-09-29T23:01:54.146 INFO:teuthology.run_tasks:Running task install... 2023-09-29T23:01:54.157 DEBUG:teuthology.task.install:project ceph 2023-09-29T23:01:54.157 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-29T23:01:54.157 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-29T23:01:54.157 INFO:teuthology.task.install:Using flavor: default 2023-09-29T23:01:54.165 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-29T23:01:54.165 INFO:teuthology.task.install:extra packages: [] 2023-09-29T23:01:54.166 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-29T23:01:54.166 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:01:54.168 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-29T23:01:54.168 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:01:54.316 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-29T23:01:54.316 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-29T23:01:54.333 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-29T23:01:54.334 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-29T23:01:54.422 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T23:01:54.422 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:01:54.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T23:01:54.451 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-29T23:01:54.452 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T23:01:54.498 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T23:01:54.499 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:01:54.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T23:01:54.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T23:01:54.527 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-29T23:01:54.527 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T23:01:54.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T23:01:54.598 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T23:01:54.664 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2023-09-29T23:01:54.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2023-09-29T23:01:54.672 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T23:01:54.739 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2023-09-29T23:01:54.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2023-09-29T23:01:54.980 INFO:teuthology.orchestra.run.smithi121.stdout:76 files removed 2023-09-29T23:01:55.005 INFO:teuthology.orchestra.run.smithi123.stdout:76 files removed 2023-09-29T23:01:55.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw 2023-09-29T23:01:55.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2023-09-29T23:01:55.704 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 293 kB/s | 78 kB 00:00 2023-09-29T23:01:55.857 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 188 kB/s | 78 kB 00:00 2023-09-29T23:01:56.072 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 28 kB/s | 9.2 kB 00:00 2023-09-29T23:01:56.124 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 40 kB/s | 9.2 kB 00:00 2023-09-29T23:01:56.339 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2023-09-29T23:01:56.416 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 5.5 kB/s | 1.5 kB 00:00 2023-09-29T23:01:57.973 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 21 MB/s | 33 MB 00:01 2023-09-29T23:01:58.166 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 19 MB/s | 33 MB 00:01 2023-09-29T23:02:15.739 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 4.1 MB/s | 48 MB 00:11 2023-09-29T23:02:15.749 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 4.1 MB/s | 48 MB 00:11 2023-09-29T23:02:23.006 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2023-09-29T23:02:23.049 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2023-09-29T23:02:23.573 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-09-29T23:02:23.741 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-09-29T23:02:24.485 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.5 MB 00:00 2023-09-29T23:02:24.699 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - PowerTools 6.7 MB/s | 6.5 MB 00:00 2023-09-29T23:02:25.935 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-09-29T23:02:26.109 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-09-29T23:02:27.085 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-09-29T23:02:27.086 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-09-29T23:02:30.920 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 392 kB/s | 24 kB 00:00 2023-09-29T23:02:30.977 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 427 kB/s | 24 kB 00:00 2023-09-29T23:02:35.281 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:02:35.283 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-29T23:02:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-29T23:02:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout:Install 21 Packages 2023-09-29T23:02:35.286 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2023-09-29T23:02:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:02:35.287 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 63 M 2023-09-29T23:02:35.287 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:02:35.368 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:02:35.370 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-29T23:02:35.371 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-29T23:02:35.372 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout:Install 21 Packages 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 2 Packages 2023-09-29T23:02:35.374 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:02:35.374 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 63 M 2023-09-29T23:02:35.374 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:02:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 5.2 MB/s | 5.2 MB 00:01 2023-09-29T23:02:36.612 INFO:teuthology.orchestra.run.smithi121.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 293 kB/s | 24 kB 00:00 2023-09-29T23:02:36.888 INFO:teuthology.orchestra.run.smithi121.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 2.8 MB/s | 796 kB 00:00 2023-09-29T23:02:37.280 INFO:teuthology.orchestra.run.smithi121.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 1.3 MB/s | 506 kB 00:00 2023-09-29T23:02:38.536 INFO:teuthology.orchestra.run.smithi123.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.8 MB/s | 5.2 MB 00:02 2023-09-29T23:02:38.619 INFO:teuthology.orchestra.run.smithi123.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 293 kB/s | 24 kB 00:00 2023-09-29T23:02:38.689 INFO:teuthology.orchestra.run.smithi121.stdout:(5/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 5.3 MB/s | 17 MB 00:03 2023-09-29T23:02:38.772 INFO:teuthology.orchestra.run.smithi121.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 537 kB/s | 45 kB 00:00 2023-09-29T23:02:38.873 INFO:teuthology.orchestra.run.smithi121.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.1 MB/s | 115 kB 00:00 2023-09-29T23:02:38.978 INFO:teuthology.orchestra.run.smithi123.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 2.2 MB/s | 796 kB 00:00 2023-09-29T23:02:38.998 INFO:teuthology.orchestra.run.smithi121.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 1.5 MB/s | 190 kB 00:00 2023-09-29T23:02:39.165 INFO:teuthology.orchestra.run.smithi121.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 2.2 MB/s | 370 kB 00:00 2023-09-29T23:02:39.228 INFO:teuthology.orchestra.run.smithi123.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 2.0 MB/s | 506 kB 00:00 2023-09-29T23:02:39.298 INFO:teuthology.orchestra.run.smithi121.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 2.6 MB/s | 358 kB 00:00 2023-09-29T23:02:39.398 INFO:teuthology.orchestra.run.smithi121.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.0 MB/s | 107 kB 00:00 2023-09-29T23:02:39.749 INFO:teuthology.orchestra.run.smithi121.stdout:(12/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 5.6 MB 00:02 2023-09-29T23:02:39.891 INFO:teuthology.orchestra.run.smithi121.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 719 kB/s | 354 kB 00:00 2023-09-29T23:02:39.983 INFO:teuthology.orchestra.run.smithi121.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 479 kB/s | 44 kB 00:00 2023-09-29T23:02:40.141 INFO:teuthology.orchestra.run.smithi121.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2023-09-29T23:02:40.167 INFO:teuthology.orchestra.run.smithi121.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 622 kB/s | 259 kB 00:00 2023-09-29T23:02:40.225 INFO:teuthology.orchestra.run.smithi121.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-09-29T23:02:40.284 INFO:teuthology.orchestra.run.smithi121.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.1 MB/s | 306 kB 00:00 2023-09-29T23:02:40.309 INFO:teuthology.orchestra.run.smithi121.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 897 kB/s | 75 kB 00:00 2023-09-29T23:02:40.385 INFO:teuthology.orchestra.run.smithi121.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 17 MB/s | 1.7 MB 00:00 2023-09-29T23:02:40.693 INFO:teuthology.orchestra.run.smithi121.stdout:(21/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 4.5 MB/s | 23 MB 00:05 2023-09-29T23:02:41.187 INFO:teuthology.orchestra.run.smithi123.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 2.8 MB/s | 5.6 MB 00:01 2023-09-29T23:02:41.303 INFO:teuthology.orchestra.run.smithi123.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 383 kB/s | 45 kB 00:00 2023-09-29T23:02:41.420 INFO:teuthology.orchestra.run.smithi123.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 990 kB/s | 115 kB 00:00 2023-09-29T23:02:41.537 INFO:teuthology.orchestra.run.smithi123.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 1.6 MB/s | 190 kB 00:00 2023-09-29T23:02:41.652 INFO:teuthology.orchestra.run.smithi121.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 2.8 MB/s | 3.6 MB 00:01 2023-09-29T23:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:(9/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 2.8 MB/s | 17 MB 00:06 2023-09-29T23:02:41.763 INFO:teuthology.orchestra.run.smithi123.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 1.6 MB/s | 370 kB 00:00 2023-09-29T23:02:41.888 INFO:teuthology.orchestra.run.smithi123.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 853 kB/s | 107 kB 00:00 2023-09-29T23:02:41.914 INFO:teuthology.orchestra.run.smithi123.stdout:(12/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 1.6 MB/s | 358 kB 00:00 2023-09-29T23:02:42.010 INFO:teuthology.orchestra.run.smithi121.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 2.0 MB/s | 3.5 MB 00:01 2023-09-29T23:02:42.013 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:02:42.013 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.4 MB/s | 63 MB 00:06 2023-09-29T23:02:42.056 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:02:42.102 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:02:42.102 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:02:42.306 INFO:teuthology.orchestra.run.smithi123.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 849 kB/s | 354 kB 00:00 2023-09-29T23:02:42.331 INFO:teuthology.orchestra.run.smithi123.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 623 kB/s | 259 kB 00:00 2023-09-29T23:02:42.415 INFO:teuthology.orchestra.run.smithi123.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 527 kB/s | 44 kB 00:00 2023-09-29T23:02:42.473 INFO:teuthology.orchestra.run.smithi123.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 331 kB/s | 47 kB 00:00 2023-09-29T23:02:42.477 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:02:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:02:42.998 INFO:teuthology.orchestra.run.smithi123.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 112 kB/s | 59 kB 00:00 2023-09-29T23:02:43.140 INFO:teuthology.orchestra.run.smithi123.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 422 kB/s | 306 kB 00:00 2023-09-29T23:02:43.182 INFO:teuthology.orchestra.run.smithi123.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 409 kB/s | 75 kB 00:00 2023-09-29T23:02:43.501 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:02:43.558 INFO:teuthology.orchestra.run.smithi123.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 4.2 MB/s | 1.7 MB 00:00 2023-09-29T23:02:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-29T23:02:43.899 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-29T23:02:44.276 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-29T23:02:44.483 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:02:44.533 INFO:teuthology.orchestra.run.smithi123.stdout:(21/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 2.6 MB/s | 3.5 MB 00:01 2023-09-29T23:02:44.559 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:02:44.833 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-29T23:02:45.007 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:02:45.308 INFO:teuthology.orchestra.run.smithi123.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 2.0 MB/s | 3.6 MB 00:01 2023-09-29T23:02:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:02:45.582 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:02:45.806 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:02:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-29T23:02:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-29T23:02:46.390 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-29T23:02:46.953 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-29T23:02:47.123 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:02:47.195 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:02:47.367 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-29T23:02:47.383 INFO:teuthology.orchestra.run.smithi123.stdout:(23/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 2.0 MB/s | 23 MB 00:11 2023-09-29T23:02:47.387 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:02:47.388 INFO:teuthology.orchestra.run.smithi123.stdout:Total 5.3 MB/s | 63 MB 00:12 2023-09-29T23:02:47.427 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:02:47.474 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:02:47.474 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:02:47.551 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:02:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:02:47.847 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:02:47.850 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:02:47.951 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-29T23:02:48.098 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:02:48.152 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:02:48.365 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-29T23:02:48.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-29T23:02:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-29T23:02:48.837 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:02:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-29T23:02:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-29T23:02:49.226 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-29T23:02:49.607 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-29T23:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:02:49.999 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:02:50.228 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-29T23:02:50.409 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:02:50.808 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:02:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:02:51.210 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-29T23:02:51.388 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-29T23:02:51.547 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-29T23:02:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:02:51.692 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-29T23:02:51.890 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:02:52.244 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-29T23:02:52.400 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:02:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:02:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:02:52.618 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-29T23:02:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:02:52.804 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:02:52.861 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:02:53.035 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:02:53.269 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-29T23:02:53.339 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:02:53.416 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:02:53.502 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-29T23:02:53.717 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-29T23:02:53.908 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-29T23:02:54.117 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-29T23:02:54.308 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-29T23:02:56.883 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:02:57.116 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:02:57.692 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-29T23:02:57.956 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:02:58.055 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-29T23:02:58.249 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:03:11.759 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:03:11.759 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2023-09-29T23:03:11.759 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2023-09-29T23:03:11.760 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-09-29T23:03:11.760 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2023-09-29T23:03:11.760 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2023-09-29T23:03:11.760 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2023-09-29T23:03:11.760 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2023-09-29T23:03:11.760 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2023-09-29T23:03:11.760 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-09-29T23:03:11.761 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:13.454 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:03:13.515 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:03:13.515 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:03:13.762 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:03:13.762 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:03:14.481 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:03:14.482 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-29T23:03:14.482 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:03:14.482 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:03:14.482 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-29T23:03:14.482 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:03:14.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-29T23:03:14.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-29T23:03:14.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-29T23:03:14.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-29T23:03:14.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:03:14.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-29T23:03:14.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:03:14.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:03:14.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-29T23:03:14.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-29T23:03:14.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-29T23:03:14.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-29T23:03:14.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-29T23:03:14.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-29T23:03:14.485 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-29T23:03:14.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-29T23:03:14.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:03:14.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-29T23:03:14.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-29T23:03:15.540 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:03:15.540 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:15.540 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2023-09-29T23:03:15.540 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.540 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.540 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-29T23:03:15.541 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-29T23:03:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:15.543 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:03:15.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /run 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2023-09-29T23:03:16.554 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2023-09-29T23:03:16.555 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2023-09-29T23:03:16.555 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-09-29T23:03:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:16.701 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:46 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:18.227 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:03:18.289 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-29T23:03:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-29T23:03:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-29T23:03:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:18.291 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:03:18.291 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:18.291 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2023-09-29T23:03:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:18.291 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 53 M 2023-09-29T23:03:18.292 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 233 M 2023-09-29T23:03:18.292 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:03:18.302 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-29T23:03:18.302 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:03:18.498 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-29T23:03:18.498 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:03:18.673 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 926 kB/s | 175 kB 00:00 2023-09-29T23:03:18.865 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): jq-1.6-7.el8.x86_64.rpm 532 kB/s | 202 kB 00:00 2023-09-29T23:03:19.023 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2023-09-29T23:03:19.049 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 499 kB/s | 187 kB 00:00 2023-09-29T23:03:19.107 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 844 kB/s | 71 kB 00:00 2023-09-29T23:03:19.182 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-29T23:03:19.183 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-29T23:03:19.184 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-29T23:03:19.185 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-29T23:03:19.185 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-29T23:03:19.185 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-29T23:03:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-29T23:03:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:20.337 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-29T23:03:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-29T23:03:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:20.340 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:03:20.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-test 2023-09-29T23:03:21.454 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:51 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:21.607 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 17 MB/s | 52 MB 00:03 2023-09-29T23:03:21.609 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:03:21.610 INFO:teuthology.orchestra.run.smithi121.stdout:Total 16 MB/s | 53 MB 00:03 2023-09-29T23:03:21.654 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:03:21.675 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:03:21.675 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:03:21.959 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:03:21.962 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:03:22.473 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:03:22.756 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-29T23:03:22.943 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-29T23:03:23.011 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:03:23.012 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:23.012 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:03:23.012 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-29T23:03:23.013 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:23.014 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:03:23.014 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:23.014 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2023-09-29T23:03:23.014 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:23.014 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 53 M 2023-09-29T23:03:23.015 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 233 M 2023-09-29T23:03:23.015 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:03:23.066 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-29T23:03:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-29T23:03:23.351 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 593 kB/s | 175 kB 00:00 2023-09-29T23:03:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-29T23:03:23.426 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): jq-1.6-7.el8.x86_64.rpm 547 kB/s | 202 kB 00:00 2023-09-29T23:03:23.557 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:03:23.602 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2023-09-29T23:03:23.702 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 707 kB/s | 71 kB 00:00 2023-09-29T23:03:23.735 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 488 kB/s | 187 kB 00:00 2023-09-29T23:03:25.769 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 19 MB/s | 52 MB 00:02 2023-09-29T23:03:25.771 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:03:25.771 INFO:teuthology.orchestra.run.smithi123.stdout:Total 19 MB/s | 53 MB 00:02 2023-09-29T23:03:25.812 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:03:25.832 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:03:25.832 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:03:26.117 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:03:26.120 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:03:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:03:26.852 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-29T23:03:27.039 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-29T23:03:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-29T23:03:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-29T23:03:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-29T23:03:27.578 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:03:28.544 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:03:28.780 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:03:29.046 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:03:29.046 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-29T23:03:29.046 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-29T23:03:29.047 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-29T23:03:29.047 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-29T23:03:29.047 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-7.el8.x86_64 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-29T23:03:29.764 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-29T23:03:29.765 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-29T23:03:29.765 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:29.765 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:03:30.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2023-09-29T23:03:30.850 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:32.441 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-29T23:03:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-29T23:03:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-29T23:03:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout:Install 32 Packages 2023-09-29T23:03:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:32.449 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 28 M 2023-09-29T23:03:32.449 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 110 M 2023-09-29T23:03:32.449 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:03:32.543 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-29T23:03:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:03:32.867 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 34 kB/s | 6.4 kB 00:00 2023-09-29T23:03:32.967 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 5.5 MB/s | 1.6 MB 00:00 2023-09-29T23:03:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-29T23:03:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-29T23:03:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-29T23:03:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-29T23:03:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-29T23:03:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-29T23:03:33.284 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 3.8 MB/s | 2.3 MB 00:00 2023-09-29T23:03:33.367 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 2.8 MB/s | 238 kB 00:00 2023-09-29T23:03:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 438 kB/s | 157 kB 00:00 2023-09-29T23:03:33.758 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-29T23:03:33.758 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:33.758 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:03:33.758 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:33.758 INFO:teuthology.orchestra.run.smithi123.stdout: jq-1.6-7.el8.x86_64 2023-09-29T23:03:33.759 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:33.759 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-29T23:03:33.759 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-29T23:03:33.759 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-29T23:03:33.759 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:33.759 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:03:33.935 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 17 MB/s | 17 MB 00:00 2023-09-29T23:03:33.960 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 166 kB/s | 39 kB 00:00 2023-09-29T23:03:33.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph 2023-09-29T23:03:34.018 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 3.4 MB/s | 3.9 MB 00:01 2023-09-29T23:03:34.052 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 619 kB/s | 57 kB 00:00 2023-09-29T23:03:34.277 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 302 kB/s | 103 kB 00:00 2023-09-29T23:03:34.303 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 993 kB/s | 249 kB 00:00 2023-09-29T23:03:34.453 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.0 MB/s | 457 kB 00:00 2023-09-29T23:03:34.503 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 544 kB/s | 109 kB 00:00 2023-09-29T23:03:34.587 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 326 kB/s | 44 kB 00:00 2023-09-29T23:03:34.646 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): python3-cryptography-3.2.1-6.el8.x86_6 1.7 MB/s | 641 kB 00:00 2023-09-29T23:03:34.671 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2023-09-29T23:03:34.705 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2023-09-29T23:03:34.730 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 204 kB/s | 11 kB 00:00 2023-09-29T23:03:34.755 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 355 kB/s | 18 kB 00:00 2023-09-29T23:03:34.781 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 973 kB/s | 49 kB 00:00 2023-09-29T23:03:34.806 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-09-29T23:03:34.832 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:34.873 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-09-29T23:03:34.898 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2023-09-29T23:03:34.923 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-09-29T23:03:34.949 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 478 kB/s | 24 kB 00:00 2023-09-29T23:03:34.974 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 580 kB/s | 29 kB 00:00 2023-09-29T23:03:34.999 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 547 kB/s | 27 kB 00:00 2023-09-29T23:03:35.025 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2023-09-29T23:03:35.050 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-09-29T23:03:35.075 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-09-29T23:03:35.100 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2023-09-29T23:03:35.125 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 851 kB/s | 85 kB 00:00 2023-09-29T23:03:35.129 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:03:35.129 INFO:teuthology.orchestra.run.smithi121.stdout:Total 11 MB/s | 28 MB 00:02 2023-09-29T23:03:35.211 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:03:35.247 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:03:35.247 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:03:35.551 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:03:35.554 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:03:36.411 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:03:36.414 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:36.414 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:03:36.414 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:36.414 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:03:36.414 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-29T23:03:36.414 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-29T23:03:36.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-29T23:03:36.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:03:36.417 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:36.418 INFO:teuthology.orchestra.run.smithi123.stdout:Install 32 Packages 2023-09-29T23:03:36.418 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:36.418 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 28 M 2023-09-29T23:03:36.418 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 110 M 2023-09-29T23:03:36.418 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:03:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:03:36.885 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-29T23:03:37.011 INFO:teuthology.orchestra.run.smithi123.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 32 kB/s | 6.4 kB 00:00 2023-09-29T23:03:37.121 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-29T23:03:37.137 INFO:teuthology.orchestra.run.smithi123.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 4.8 MB/s | 1.6 MB 00:00 2023-09-29T23:03:37.187 INFO:teuthology.orchestra.run.smithi123.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 6.1 MB/s | 2.3 MB 00:00 2023-09-29T23:03:37.270 INFO:teuthology.orchestra.run.smithi123.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 2.8 MB/s | 238 kB 00:00 2023-09-29T23:03:37.345 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-29T23:03:37.612 INFO:teuthology.orchestra.run.smithi123.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 460 kB/s | 157 kB 00:00 2023-09-29T23:03:37.636 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-29T23:03:37.696 INFO:teuthology.orchestra.run.smithi123.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 465 kB/s | 39 kB 00:00 2023-09-29T23:03:37.858 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-29T23:03:37.938 INFO:teuthology.orchestra.run.smithi123.stdout:(7/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 21 MB/s | 17 MB 00:00 2023-09-29T23:03:37.963 INFO:teuthology.orchestra.run.smithi123.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 387 kB/s | 103 kB 00:00 2023-09-29T23:03:38.030 INFO:teuthology.orchestra.run.smithi123.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 623 kB/s | 57 kB 00:00 2023-09-29T23:03:38.080 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-29T23:03:38.247 INFO:teuthology.orchestra.run.smithi123.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 249 kB 00:00 2023-09-29T23:03:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-29T23:03:38.414 INFO:teuthology.orchestra.run.smithi123.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.0 MB/s | 457 kB 00:00 2023-09-29T23:03:38.477 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-29T23:03:38.490 INFO:teuthology.orchestra.run.smithi123.stdout:(12/32): python3-cryptography-3.2.1-6.el8.x86_6 2.6 MB/s | 641 kB 00:00 2023-09-29T23:03:38.548 INFO:teuthology.orchestra.run.smithi123.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 748 kB/s | 44 kB 00:00 2023-09-29T23:03:38.590 INFO:teuthology.orchestra.run.smithi123.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 620 kB/s | 109 kB 00:00 2023-09-29T23:03:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.7 MB/s | 185 kB 00:00 2023-09-29T23:03:38.666 INFO:teuthology.orchestra.run.smithi123.stdout:(16/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 2.4 MB/s | 3.9 MB 00:01 2023-09-29T23:03:38.678 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-29T23:03:38.691 INFO:teuthology.orchestra.run.smithi123.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2023-09-29T23:03:38.725 INFO:teuthology.orchestra.run.smithi123.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 530 kB/s | 18 kB 00:00 2023-09-29T23:03:38.771 INFO:teuthology.orchestra.run.smithi123.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 106 kB/s | 11 kB 00:00 2023-09-29T23:03:38.800 INFO:teuthology.orchestra.run.smithi123.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 648 kB/s | 49 kB 00:00 2023-09-29T23:03:38.825 INFO:teuthology.orchestra.run.smithi123.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2023-09-29T23:03:38.851 INFO:teuthology.orchestra.run.smithi123.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2023-09-29T23:03:38.876 INFO:teuthology.orchestra.run.smithi123.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-09-29T23:03:38.901 INFO:teuthology.orchestra.run.smithi123.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-09-29T23:03:38.906 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-29T23:03:38.927 INFO:teuthology.orchestra.run.smithi123.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2023-09-29T23:03:38.952 INFO:teuthology.orchestra.run.smithi123.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 585 kB/s | 29 kB 00:00 2023-09-29T23:03:38.977 INFO:teuthology.orchestra.run.smithi123.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 550 kB/s | 27 kB 00:00 2023-09-29T23:03:39.002 INFO:teuthology.orchestra.run.smithi123.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-09-29T23:03:39.028 INFO:teuthology.orchestra.run.smithi123.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-09-29T23:03:39.053 INFO:teuthology.orchestra.run.smithi123.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-09-29T23:03:39.078 INFO:teuthology.orchestra.run.smithi123.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 460 kB/s | 23 kB 00:00 2023-09-29T23:03:39.103 INFO:teuthology.orchestra.run.smithi123.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2023-09-29T23:03:39.105 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:03:39.105 INFO:teuthology.orchestra.run.smithi123.stdout:Total 11 MB/s | 28 MB 00:02 2023-09-29T23:03:39.150 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-29T23:03:39.195 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:03:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:03:39.235 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:03:39.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-29T23:03:39.529 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:03:39.532 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:03:39.798 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-29T23:03:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-29T23:03:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-29T23:03:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-29T23:03:40.629 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:03:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-29T23:03:40.844 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-29T23:03:41.077 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-29T23:03:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-29T23:03:41.315 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-29T23:03:41.351 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-29T23:03:41.489 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-29T23:03:41.583 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-29T23:03:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-29T23:03:41.780 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-29T23:03:41.927 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-29T23:03:41.979 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-29T23:03:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-29T23:03:42.220 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-29T23:03:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-29T23:03:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-29T23:03:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-29T23:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-29T23:03:42.836 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-29T23:03:43.060 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-29T23:03:43.100 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-29T23:03:43.322 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-29T23:03:43.536 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-29T23:03:43.718 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-29T23:03:43.764 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-29T23:03:43.995 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-29T23:03:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-29T23:03:44.504 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-29T23:03:44.710 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-29T23:03:45.031 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-29T23:03:45.248 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-29T23:03:45.252 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-29T23:03:45.453 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-29T23:03:45.472 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-29T23:03:45.668 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-29T23:03:45.866 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-29T23:03:45.931 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-29T23:03:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-29T23:03:46.180 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-29T23:03:46.286 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-29T23:03:46.408 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-29T23:03:46.494 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-29T23:03:46.512 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-29T23:03:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-29T23:03:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-29T23:03:47.086 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-29T23:03:47.333 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-29T23:03:47.333 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-29T23:03:47.333 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-29T23:03:47.334 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-29T23:03:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-29T23:03:47.336 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-29T23:03:47.336 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-29T23:03:47.336 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-29T23:03:47.336 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-29T23:03:47.336 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-29T23:03:47.337 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-29T23:03:47.445 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-29T23:03:47.673 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:48.244 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-29T23:03:48.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-29T23:03:48.246 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-29T23:03:48.247 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-29T23:03:48.247 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-29T23:03:48.247 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-29T23:03:48.247 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-29T23:03:48.247 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:48.247 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:03:48.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2023-09-29T23:03:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-29T23:03:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-29T23:03:49.401 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:49.748 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-29T23:03:49.949 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-29T23:03:50.165 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-29T23:03:50.341 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-29T23:03:50.359 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-29T23:03:50.499 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-29T23:03:50.789 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:03:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-29T23:03:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-29T23:03:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-29T23:03:50.896 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-29T23:03:50.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-29T23:03:50.978 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:03:50.979 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:03:50.979 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:03:51.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2023-09-29T23:03:51.938 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:03:52.404 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-29T23:03:52.405 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-2.6-1.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-29T23:03:52.406 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-29T23:03:52.407 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-29T23:03:52.407 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-29T23:03:52.407 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-29T23:03:52.407 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-29T23:03:52.407 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-29T23:03:52.407 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:52.407 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:03:52.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base 2023-09-29T23:03:53.506 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:03:53.506 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:53.506 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:03:53.506 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:53.507 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:03:53.507 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-29T23:03:53.507 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:53.507 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:03:53.507 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:53.507 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:03:53.507 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:53.508 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 80 k 2023-09-29T23:03:53.508 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 363 k 2023-09-29T23:03:53.508 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:03:53.531 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:53.758 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 320 kB/s | 80 kB 00:00 2023-09-29T23:03:53.759 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:03:53.759 INFO:teuthology.orchestra.run.smithi121.stdout:Total 319 kB/s | 80 kB 00:00 2023-09-29T23:03:53.759 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:03:53.767 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:03:53.767 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:03:53.768 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:03:53.772 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:03:53.811 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:03:54.308 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:54.940 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:03:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:03:55.132 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:03:55.132 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:03:55.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install cephadm 2023-09-29T23:03:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:55.308 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:03:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:03:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:55.308 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:03:55.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2023-09-29T23:03:56.078 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:56.357 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:03:57.657 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:03:57.657 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:57.657 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:03:57.657 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:57.657 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:03:57.658 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-29T23:03:57.658 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:57.658 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:03:57.658 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:03:57.658 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:03:57.658 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:57.659 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 80 k 2023-09-29T23:03:57.659 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 363 k 2023-09-29T23:03:57.659 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:03:57.818 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 502 kB/s | 80 kB 00:00 2023-09-29T23:03:57.819 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:03:57.819 INFO:teuthology.orchestra.run.smithi123.stdout:Total 498 kB/s | 80 kB 00:00 2023-09-29T23:03:57.819 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:03:57.827 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:03:57.827 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:03:57.829 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:03:57.832 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:03:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:03:57.946 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:03:57.947 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-29T23:03:57.947 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:57.947 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:03:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-29T23:03:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:57.948 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:03:57.948 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:03:57.948 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:03:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:03:57.948 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 161 k 2023-09-29T23:03:57.949 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 490 k 2023-09-29T23:03:57.949 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:03:58.322 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:58.550 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:58.779 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:58.864 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 176 kB/s | 161 kB 00:00 2023-09-29T23:03:58.865 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:03:58.865 INFO:teuthology.orchestra.run.smithi121.stdout:Total 176 kB/s | 161 kB 00:00 2023-09-29T23:03:58.866 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:03:58.877 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:03:58.878 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:03:58.929 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:03:58.933 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:03:59.395 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:03:59.395 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:59.395 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:03:59.395 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:03:59.395 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:03:59.396 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:03:59.497 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:03:59.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-immutable-object-cache 2023-09-29T23:03:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:04:00.142 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:04:00.456 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:00.766 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:04:00.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:00.766 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:00.766 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:04:00.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:00.766 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:00.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2023-09-29T23:04:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:02.035 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:02.035 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:02.035 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2023-09-29T23:04:02.035 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:02.036 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:02.036 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-29T23:04:02.036 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:02.036 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:02.036 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:02.036 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:04:02.036 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:02.037 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 161 k 2023-09-29T23:04:02.037 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 490 k 2023-09-29T23:04:02.037 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:02.230 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 834 kB/s | 161 kB 00:00 2023-09-29T23:04:02.231 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:02.231 INFO:teuthology.orchestra.run.smithi123.stdout:Total 829 kB/s | 161 kB 00:00 2023-09-29T23:04:02.231 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:02.242 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:02.243 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:02.289 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:02.292 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:02.401 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:03.198 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:04:03.251 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:04:03.440 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:03.441 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:04:03.442 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:03.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2023-09-29T23:04:03.856 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:04:04.389 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:05.289 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-29T23:04:05.289 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:05.289 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:05.290 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:04:05.290 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:05.290 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:05.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr 2023-09-29T23:04:05.977 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:05.978 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:04:05.978 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:04:05.979 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:05.980 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2023-09-29T23:04:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:05.980 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.0 M 2023-09-29T23:04:05.980 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 8.0 M 2023-09-29T23:04:05.980 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:04:06.404 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:06.514 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 97 kB/s | 14 kB 00:00 2023-09-29T23:04:06.540 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 132 kB/s | 23 kB 00:00 2023-09-29T23:04:06.665 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 5.7 MB/s | 1.7 MB 00:00 2023-09-29T23:04:06.690 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2023-09-29T23:04:06.974 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 77 kB/s | 34 kB 00:00 2023-09-29T23:04:07.266 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 329 kB/s | 196 kB 00:00 2023-09-29T23:04:07.269 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:07.269 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-09-29T23:04:07.281 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:04:07.291 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:04:07.291 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:04:07.375 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:04:07.378 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:04:07.636 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:04:07.788 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:04:07.853 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-29T23:04:07.977 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:07.981 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:04:07.982 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:08.081 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-dashboard 2023-09-29T23:04:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-29T23:04:08.292 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-29T23:04:08.388 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-29T23:04:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-29T23:04:08.931 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:08.974 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:04:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:04:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-29T23:04:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-29T23:04:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-29T23:04:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-29T23:04:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-29T23:04:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-29T23:04:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:09.902 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:09.902 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-29T23:04:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-29T23:04:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-29T23:04:09.903 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:09.903 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:10.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-29T23:04:10.500 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:10.501 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:10.501 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:04:10.501 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:10.501 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:10.501 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-29T23:04:10.501 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:04:10.501 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2023-09-29T23:04:10.502 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:10.503 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 2.0 M 2023-09-29T23:04:10.503 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 8.0 M 2023-09-29T23:04:10.503 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:10.733 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 153 kB/s | 23 kB 00:00 2023-09-29T23:04:10.834 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 58 kB/s | 14 kB 00:00 2023-09-29T23:04:10.876 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 5.8 MB/s | 1.7 MB 00:00 2023-09-29T23:04:10.901 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2023-09-29T23:04:10.993 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2023-09-29T23:04:11.101 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 872 kB/s | 196 kB 00:00 2023-09-29T23:04:11.103 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:11.103 INFO:teuthology.orchestra.run.smithi123.stdout:Total 3.3 MB/s | 2.0 MB 00:00 2023-09-29T23:04:11.113 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:11.122 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:11.122 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:11.166 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:11.198 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:11.201 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:11.676 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-29T23:04:11.946 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-29T23:04:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-29T23:04:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-29T23:04:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-29T23:04:12.758 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:12.759 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================================ 2023-09-29T23:04:12.759 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:04:12.759 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================================ 2023-09-29T23:04:12.759 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:04:12.759 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:04:12.760 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================================ 2023-09-29T23:04:12.761 INFO:teuthology.orchestra.run.smithi121.stdout:Install 8 Packages 2023-09-29T23:04:12.761 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:12.761 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 35 M 2023-09-29T23:04:12.761 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 210 M 2023-09-29T23:04:12.761 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:04:12.851 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:04:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-29T23:04:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-29T23:04:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-29T23:04:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-29T23:04:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-29T23:04:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-29T23:04:13.715 INFO:teuthology.orchestra.run.smithi121.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-116 10 MB/s | 7.4 MB 00:00 2023-09-29T23:04:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-29T23:04:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:13.772 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-29T23:04:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-29T23:04:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-29T23:04:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:13.773 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:13.799 INFO:teuthology.orchestra.run.smithi121.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.0 MB/s | 4.7 MB 00:00 2023-09-29T23:04:13.865 INFO:teuthology.orchestra.run.smithi121.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2023-09-29T23:04:13.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-29T23:04:13.949 INFO:teuthology.orchestra.run.smithi121.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.6 MB/s | 225 kB 00:00 2023-09-29T23:04:14.408 INFO:teuthology.orchestra.run.smithi121.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 25 MB/s | 14 MB 00:00 2023-09-29T23:04:14.433 INFO:teuthology.orchestra.run.smithi121.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 645 kB 00:00 2023-09-29T23:04:14.583 INFO:teuthology.orchestra.run.smithi121.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.3 MB/s | 3.7 MB 00:00 2023-09-29T23:04:14.642 INFO:teuthology.orchestra.run.smithi121.stdout:(8/8): libquadmath-8.5.0-20.el8.x86_64.rpm 738 kB/s | 172 kB 00:00 2023-09-29T23:04:14.642 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:14.642 INFO:teuthology.orchestra.run.smithi121.stdout:Total 19 MB/s | 35 MB 00:01 2023-09-29T23:04:14.751 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:14.875 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:04:14.893 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:04:14.893 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:04:15.286 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:04:15.289 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:04:16.342 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:16.342 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================================ 2023-09-29T23:04:16.342 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================================ 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-29T23:04:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================================ 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout:Install 8 Packages 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 35 M 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 210 M 2023-09-29T23:04:16.344 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:16.890 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:04:17.070 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-29T23:04:17.117 INFO:teuthology.orchestra.run.smithi123.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-116 11 MB/s | 7.4 MB 00:00 2023-09-29T23:04:17.199 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-29T23:04:17.201 INFO:teuthology.orchestra.run.smithi123.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.1 MB/s | 4.7 MB 00:00 2023-09-29T23:04:17.308 INFO:teuthology.orchestra.run.smithi123.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.5 MB/s | 4.9 MB 00:00 2023-09-29T23:04:17.335 INFO:teuthology.orchestra.run.smithi123.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2023-09-29T23:04:17.345 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-29T23:04:17.843 INFO:teuthology.orchestra.run.smithi123.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2023-09-29T23:04:17.902 INFO:teuthology.orchestra.run.smithi123.stdout:(6/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.8 MB/s | 3.7 MB 00:00 2023-09-29T23:04:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-29T23:04:18.077 INFO:teuthology.orchestra.run.smithi123.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 738 kB/s | 172 kB 00:00 2023-09-29T23:04:18.102 INFO:teuthology.orchestra.run.smithi123.stdout:(8/8): libgfortran-8.5.0-20.el8.x86_64.rpm 841 kB/s | 645 kB 00:00 2023-09-29T23:04:18.103 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:18.103 INFO:teuthology.orchestra.run.smithi123.stdout:Total 20 MB/s | 35 MB 00:01 2023-09-29T23:04:18.162 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-29T23:04:18.334 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:18.352 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:18.352 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:18.744 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:18.747 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:18.783 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-29T23:04:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-29T23:04:19.290 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:19.436 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-29T23:04:19.569 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-29T23:04:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-29T23:04:19.711 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-29T23:04:19.932 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-29T23:04:20.203 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-29T23:04:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-29T23:04:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-29T23:04:21.711 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-29T23:04:21.869 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-29T23:04:21.959 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-29T23:04:22.565 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-29T23:04:22.897 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-29T23:04:23.127 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-29T23:04:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-29T23:04:23.411 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:04:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:04:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-29T23:04:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-29T23:04:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-29T23:04:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-29T23:04:23.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-29T23:04:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-29T23:04:23.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-29T23:04:24.370 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-29T23:04:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-29T23:04:24.372 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:24.372 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:24.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2023-09-29T23:04:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-29T23:04:25.339 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:26.142 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-29T23:04:26.360 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:04:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-29T23:04:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-29T23:04:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-29T23:04:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-29T23:04:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-29T23:04:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-29T23:04:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-29T23:04:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-29T23:04:26.925 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:26.927 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:26.927 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:04:26.927 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:26.927 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:04:26.927 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-29T23:04:26.928 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-29T23:04:26.929 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-29T23:04:26.929 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-29T23:04:26.929 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:26.929 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:04:26.929 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:26.929 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2023-09-29T23:04:26.929 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:26.930 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.9 M 2023-09-29T23:04:26.930 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 25 M 2023-09-29T23:04:26.930 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:04:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-29T23:04:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:27.329 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-29T23:04:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-29T23:04:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-29T23:04:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-29T23:04:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-29T23:04:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-29T23:04:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-29T23:04:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:27.330 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:27.393 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 282 kB/s | 47 kB 00:00 2023-09-29T23:04:27.418 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 103 kB/s | 20 kB 00:00 2023-09-29T23:04:27.435 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 126 kB/s | 27 kB 00:00 2023-09-29T23:04:27.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-rook 2023-09-29T23:04:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 527 kB/s | 110 kB 00:00 2023-09-29T23:04:27.744 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 360 kB/s | 126 kB 00:00 2023-09-29T23:04:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 463 kB/s | 155 kB 00:00 2023-09-29T23:04:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 222 kB/s | 43 kB 00:00 2023-09-29T23:04:27.912 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 197 kB/s | 33 kB 00:00 2023-09-29T23:04:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 102 kB/s | 16 kB 00:00 2023-09-29T23:04:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 488 kB/s | 81 kB 00:00 2023-09-29T23:04:28.012 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 769 kB/s | 64 kB 00:00 2023-09-29T23:04:28.037 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-09-29T23:04:28.254 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.4 MB/s | 1.2 MB 00:00 2023-09-29T23:04:28.257 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:28.257 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2023-09-29T23:04:28.301 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:28.318 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:04:28.334 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:04:28.334 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:04:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:04:28.424 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:04:28.975 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:04:29.212 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-29T23:04:29.410 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-29T23:04:29.620 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-29T23:04:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-29T23:04:29.870 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:29.871 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:29.872 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:04:29.872 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:29.872 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:29.872 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-29T23:04:29.872 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:04:29.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-29T23:04:29.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-29T23:04:29.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-29T23:04:29.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-29T23:04:29.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-29T23:04:29.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-29T23:04:29.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-29T23:04:29.874 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-29T23:04:29.874 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-29T23:04:29.874 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-29T23:04:29.874 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-29T23:04:29.875 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-29T23:04:29.875 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:29.876 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:29.876 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:29.876 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2023-09-29T23:04:29.876 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:29.878 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 1.9 M 2023-09-29T23:04:29.878 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 25 M 2023-09-29T23:04:29.878 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:30.065 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-29T23:04:30.280 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-29T23:04:30.409 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 295 kB/s | 47 kB 00:00 2023-09-29T23:04:30.434 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 108 kB/s | 20 kB 00:00 2023-09-29T23:04:30.451 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 131 kB/s | 27 kB 00:00 2023-09-29T23:04:30.572 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-29T23:04:30.635 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 549 kB/s | 110 kB 00:00 2023-09-29T23:04:30.662 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 735 kB/s | 155 kB 00:00 2023-09-29T23:04:30.737 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 385 kB/s | 126 kB 00:00 2023-09-29T23:04:30.788 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 279 kB/s | 43 kB 00:00 2023-09-29T23:04:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-29T23:04:30.813 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 219 kB/s | 33 kB 00:00 2023-09-29T23:04:30.871 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2023-09-29T23:04:30.930 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2023-09-29T23:04:30.955 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 487 kB/s | 81 kB 00:00 2023-09-29T23:04:30.980 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-09-29T23:04:31.206 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.0 MB/s | 1.2 MB 00:00 2023-09-29T23:04:31.208 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:31.208 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-09-29T23:04:31.266 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:31.281 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:31.281 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:31.367 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:31.370 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:31.860 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-29T23:04:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-29T23:04:32.219 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-29T23:04:32.324 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-29T23:04:32.425 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-29T23:04:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-29T23:04:32.662 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-29T23:04:32.667 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:04:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-29T23:04:32.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-29T23:04:32.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-29T23:04:32.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-29T23:04:32.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-29T23:04:32.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-29T23:04:32.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-29T23:04:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-29T23:04:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-29T23:04:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-29T23:04:33.657 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-29T23:04:33.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:33.657 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:33.657 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-29T23:04:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-29T23:04:33.659 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-29T23:04:33.659 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:33.659 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:33.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2023-09-29T23:04:33.829 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-29T23:04:34.610 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-29T23:04:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-29T23:04:35.374 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-29T23:04:35.575 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-29T23:04:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:04:36.038 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-29T23:04:36.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-29T23:04:36.040 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-29T23:04:36.040 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-29T23:04:36.040 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-29T23:04:36.178 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:36.179 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:04:36.179 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:36.180 INFO:teuthology.orchestra.run.smithi121.stdout:Install 5 Packages 2023-09-29T23:04:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:36.181 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.0 M 2023-09-29T23:04:36.181 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2023-09-29T23:04:36.181 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:04:36.751 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-29T23:04:36.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:36.751 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:36.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:36.751 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-29T23:04:36.752 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-29T23:04:36.753 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:36.753 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:36.754 INFO:teuthology.orchestra.run.smithi121.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 663 kB/s | 121 kB 00:00 2023-09-29T23:04:36.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-cephadm 2023-09-29T23:04:37.013 INFO:teuthology.orchestra.run.smithi121.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.8 MB/s | 476 kB 00:00 2023-09-29T23:04:37.038 INFO:teuthology.orchestra.run.smithi121.stdout:(3/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.1 MB/s | 538 kB 00:00 2023-09-29T23:04:37.063 INFO:teuthology.orchestra.run.smithi121.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 1.2 MB/s | 60 kB 00:00 2023-09-29T23:04:37.337 INFO:teuthology.orchestra.run.smithi121.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 6.3 MB/s | 4.8 MB 00:00 2023-09-29T23:04:37.339 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:37.340 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.1 MB/s | 6.0 MB 00:01 2023-09-29T23:04:37.396 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:04:37.408 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:04:37.408 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:04:37.519 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:04:37.522 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:04:37.745 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:37.783 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:04:38.098 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-29T23:04:39.051 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-29T23:04:39.332 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-29T23:04:39.333 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-29T23:04:39.334 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-29T23:04:39.334 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-29T23:04:39.334 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:39.334 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:39.334 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:39.334 INFO:teuthology.orchestra.run.smithi123.stdout:Install 5 Packages 2023-09-29T23:04:39.334 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:39.335 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 6.0 M 2023-09-29T23:04:39.335 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 26 M 2023-09-29T23:04:39.335 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:39.448 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-29T23:04:39.693 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-29T23:04:39.738 INFO:teuthology.orchestra.run.smithi123.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 650 kB/s | 121 kB 00:00 2023-09-29T23:04:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:04:40.005 INFO:teuthology.orchestra.run.smithi123.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2023-09-29T23:04:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:04:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-29T23:04:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-29T23:04:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-29T23:04:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-29T23:04:40.214 INFO:teuthology.orchestra.run.smithi123.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 7.2 MB/s | 4.8 MB 00:00 2023-09-29T23:04:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-29T23:04:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:40.795 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-29T23:04:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-29T23:04:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-29T23:04:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-29T23:04:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:40.796 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:40.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2023-09-29T23:04:41.072 INFO:teuthology.orchestra.run.smithi123.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 56 kB/s | 60 kB 00:01 2023-09-29T23:04:41.464 INFO:teuthology.orchestra.run.smithi123.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 276 kB/s | 476 kB 00:01 2023-09-29T23:04:41.466 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:41.466 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.8 MB/s | 6.0 MB 00:02 2023-09-29T23:04:41.522 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:41.533 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:41.533 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:41.780 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:42.223 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-29T23:04:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-29T23:04:43.364 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:43.364 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2023-09-29T23:04:43.365 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:43.366 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 957 k 2023-09-29T23:04:43.366 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 3.0 M 2023-09-29T23:04:43.366 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:04:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-29T23:04:43.756 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 421 kB/s | 83 kB 00:00 2023-09-29T23:04:43.839 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 3.0 MB/s | 874 kB 00:00 2023-09-29T23:04:43.839 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:43.840 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.0 MB/s | 957 kB 00:00 2023-09-29T23:04:43.842 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-29T23:04:43.850 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:04:43.871 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:04:43.872 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:04:43.957 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:04:43.960 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:04:44.040 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:04:44.124 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:04:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-29T23:04:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-29T23:04:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-29T23:04:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-29T23:04:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-29T23:04:44.402 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-29T23:04:44.606 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:04:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-29T23:04:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-29T23:04:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-29T23:04:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-29T23:04:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-29T23:04:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:04:45.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-29T23:04:45.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-fuse 2023-09-29T23:04:45.615 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-29T23:04:45.615 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:45.615 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:45.615 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-29T23:04:45.616 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:45.616 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:45.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2023-09-29T23:04:45.929 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:46.702 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:47.509 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:47.510 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:47.510 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:04:47.510 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2023-09-29T23:04:47.511 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:47.512 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 957 k 2023-09-29T23:04:47.512 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 3.0 M 2023-09-29T23:04:47.512 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:47.706 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 543 kB/s | 83 kB 00:00 2023-09-29T23:04:47.806 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 3.4 MB/s | 874 kB 00:00 2023-09-29T23:04:47.806 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:47.807 INFO:teuthology.orchestra.run.smithi123.stdout:Total 3.2 MB/s | 957 kB 00:00 2023-09-29T23:04:47.817 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:47.838 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:47.838 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:47.923 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:47.926 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:48.105 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:48.298 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:48.299 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:48.299 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-29T23:04:48.299 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:48.299 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:04:48.299 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 267 k 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.3 M 2023-09-29T23:04:48.300 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:04:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-29T23:04:48.565 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:04:48.626 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 819 kB/s | 267 kB 00:00 2023-09-29T23:04:48.627 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:48.627 INFO:teuthology.orchestra.run.smithi121.stdout:Total 816 kB/s | 267 kB 00:00 2023-09-29T23:04:48.627 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:04:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:04:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:04:48.675 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:04:48.678 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:04:48.856 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:04:48.954 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-29T23:04:48.954 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-29T23:04:49.131 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:04:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:04:49.609 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-29T23:04:49.609 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:49.609 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:49.610 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-29T23:04:49.610 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:49.610 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:49.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-volume 2023-09-29T23:04:50.154 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:04:50.154 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:50.155 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:50.155 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:50.155 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:50.155 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:50.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2023-09-29T23:04:50.640 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:51.223 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:52.227 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:52.228 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:52.229 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:04:52.229 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:52.229 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 267 k 2023-09-29T23:04:52.229 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 1.3 M 2023-09-29T23:04:52.229 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:52.450 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.2 MB/s | 267 kB 00:00 2023-09-29T23:04:52.451 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:52.451 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.2 MB/s | 267 kB 00:00 2023-09-29T23:04:52.451 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:52.461 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:52.461 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:52.495 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:52.498 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:52.676 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:52.800 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:52.800 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:52.800 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-29T23:04:52.800 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:52.800 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:04:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-29T23:04:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:52.801 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:04:52.801 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:04:52.801 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:04:52.801 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:52.802 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 130 k 2023-09-29T23:04:52.802 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 464 k 2023-09-29T23:04:52.802 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:04:53.400 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 217 kB/s | 130 kB 00:00 2023-09-29T23:04:53.401 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:53.401 INFO:teuthology.orchestra.run.smithi121.stdout:Total 216 kB/s | 130 kB 00:00 2023-09-29T23:04:53.402 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:04:53.408 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:04:53.408 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:04:53.464 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:04:53.469 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:04:53.981 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:04:54.226 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:04:54.493 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:04:55.027 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:04:55.120 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-29T23:04:55.120 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:55.120 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:55.121 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-29T23:04:55.121 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:55.121 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:55.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados-devel 2023-09-29T23:04:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:04:55.942 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:04:55.942 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:55.943 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:04:55.943 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:04:55.943 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:04:55.943 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:56.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2023-09-29T23:04:56.179 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:57.034 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:57.743 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:04:57.744 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2023-09-29T23:04:57.744 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:57.745 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:04:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-29T23:04:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:57.745 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:04:57.745 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:04:57.745 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:04:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:57.746 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 130 k 2023-09-29T23:04:57.746 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 464 k 2023-09-29T23:04:57.746 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:04:57.931 INFO:teuthology.orchestra.run.smithi123.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 700 kB/s | 130 kB 00:00 2023-09-29T23:04:57.932 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:04:57.932 INFO:teuthology.orchestra.run.smithi123.stdout:Total 695 kB/s | 130 kB 00:00 2023-09-29T23:04:57.932 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:04:57.937 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:04:57.937 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:04:57.988 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:04:57.991 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:04:58.099 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:04:58.324 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:04:58.433 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:04:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:04:58.623 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:04:58.625 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:04:58.625 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:04:58.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2023-09-29T23:04:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:04:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:59.254 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:04:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:04:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:04:59.254 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:04:59.583 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:04:59.645 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs2 2023-09-29T23:05:00.507 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:01.165 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-29T23:05:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:05:01.165 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-29T23:05:01.166 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:05:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:05:01.166 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 25 k 2023-09-29T23:05:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 109 k 2023-09-29T23:05:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:05:01.405 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 104 kB/s | 25 kB 00:00 2023-09-29T23:05:01.406 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:01.406 INFO:teuthology.orchestra.run.smithi121.stdout:Total 104 kB/s | 25 kB 00:00 2023-09-29T23:05:01.406 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:05:01.409 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:05:01.409 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:05:01.411 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:05:01.414 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:05:01.466 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:05:01.668 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:01.894 INFO:teuthology.orchestra.run.smithi123.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:01.917 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:02.083 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:02.084 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:05:02.084 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:02.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs-devel 2023-09-29T23:05:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:02.495 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:02.495 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:05:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:02.496 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:02.496 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:02.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2023-09-29T23:05:03.032 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:03.488 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:04.609 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:05:04.610 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:04.611 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:05:04.611 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:04.611 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 25 k 2023-09-29T23:05:04.611 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 109 k 2023-09-29T23:05:04.611 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:05:04.752 INFO:teuthology.orchestra.run.smithi123.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 176 kB/s | 25 kB 00:00 2023-09-29T23:05:04.753 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:04.753 INFO:teuthology.orchestra.run.smithi123.stdout:Total 175 kB/s | 25 kB 00:00 2023-09-29T23:05:04.753 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:05:04.756 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:05:04.756 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:05:04.759 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:05:04.762 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:05:04.814 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:05:04.875 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:05.019 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:05.067 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:05.068 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:05:05.068 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:05.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2023-09-29T23:05:05.281 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:05.960 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:05.960 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:05.961 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:05:05.961 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:05.961 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:05.961 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:06.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados2 2023-09-29T23:05:06.086 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:06.921 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:07.474 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:07.664 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:07.665 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:05:07.665 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:07.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2023-09-29T23:05:08.314 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:08.508 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:08.509 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:05:08.509 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:08.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librbd1 2023-09-29T23:05:08.603 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:09.450 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:39 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:10.011 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:10.197 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:10.198 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:05:10.198 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:10.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2023-09-29T23:05:10.827 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:11.020 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:11.021 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:05:11.021 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:11.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rados 2023-09-29T23:05:11.156 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:11.974 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:12.556 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:12.743 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:05:12.744 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:12.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2023-09-29T23:05:13.370 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:13.556 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:13.557 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:05:13.557 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:13.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rgw 2023-09-29T23:05:13.690 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:43 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:14.500 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:44 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:15.088 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:15.276 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:15.277 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:05:15.278 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:15.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2023-09-29T23:05:15.887 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:16.072 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:16.073 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:05:16.073 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:16.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-cephfs 2023-09-29T23:05:16.230 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:46 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:17.029 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:47 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:17.622 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:17.813 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:17.814 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-29T23:05:17.814 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:17.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2023-09-29T23:05:18.426 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:18.612 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:18.613 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:05:18.614 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:18.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rbd 2023-09-29T23:05:18.791 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:48 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:19.566 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:49 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:20.366 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:20.366 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:20.367 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-09-29T23:05:20.367 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:20.367 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:05:20.367 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-29T23:05:20.367 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:20.367 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:05:20.367 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:20.368 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:05:20.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:20.368 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 90 k 2023-09-29T23:05:20.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 264 k 2023-09-29T23:05:20.369 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:05:20.526 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 570 kB/s | 90 kB 00:00 2023-09-29T23:05:20.527 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:20.527 INFO:teuthology.orchestra.run.smithi121.stdout:Total 566 kB/s | 90 kB 00:00 2023-09-29T23:05:20.527 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:05:20.537 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:05:20.537 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:05:20.585 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:05:20.588 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:05:20.697 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:05:20.918 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-29T23:05:21.130 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:21.131 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-09-29T23:05:21.131 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:21.207 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:21.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-fuse 2023-09-29T23:05:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:22.062 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:05:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:22.063 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:22.080 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:52 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:22.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2023-09-29T23:05:23.135 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:53 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:23.662 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:23.662 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:23.662 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:05:23.663 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:23.664 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 90 k 2023-09-29T23:05:23.664 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 264 k 2023-09-29T23:05:23.664 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:05:23.833 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 534 kB/s | 90 kB 00:00 2023-09-29T23:05:23.833 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:23.834 INFO:teuthology.orchestra.run.smithi123.stdout:Total 530 kB/s | 90 kB 00:00 2023-09-29T23:05:23.834 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:05:23.844 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:05:23.844 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:05:23.894 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:05:23.897 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:05:23.997 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:05:24.225 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:24.511 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:24.718 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:24.718 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:24.718 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-29T23:05:24.718 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:24.718 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.5 M 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 16 M 2023-09-29T23:05:24.719 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:05:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:25.130 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:05:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:25.131 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:25.246 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 6.5 MB/s | 3.5 MB 00:00 2023-09-29T23:05:25.246 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:25.246 INFO:teuthology.orchestra.run.smithi121.stdout:Total 6.5 MB/s | 3.5 MB 00:00 2023-09-29T23:05:25.247 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:05:25.259 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:05:25.259 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:05:25.321 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:05:25.323 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:05:25.339 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-mirror 2023-09-29T23:05:25.762 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:05:25.931 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:26.195 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:56 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:26.994 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:26.994 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:26.994 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:05:26.995 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:26.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:26.995 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:27.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2023-09-29T23:05:27.751 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:27.752 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2023-09-29T23:05:27.752 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.5 M 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 16 M 2023-09-29T23:05:27.753 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:05:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:58 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:28.360 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 5.7 MB/s | 3.5 MB 00:00 2023-09-29T23:05:28.361 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:28.361 INFO:teuthology.orchestra.run.smithi123.stdout:Total 5.7 MB/s | 3.5 MB 00:00 2023-09-29T23:05:28.362 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:05:28.373 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:05:28.373 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:05:28.435 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:05:28.438 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:05:28.877 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:05:29.059 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-29T23:05:29.699 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-29T23:05:29.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 173 k 2023-09-29T23:05:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 552 k 2023-09-29T23:05:29.701 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-29T23:05:29.890 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 915 kB/s | 173 kB 00:00 2023-09-29T23:05:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Total 909 kB/s | 173 kB 00:00 2023-09-29T23:05:29.891 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-29T23:05:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-29T23:05:29.902 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-29T23:05:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-29T23:05:29.953 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-29T23:05:30.099 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-29T23:05:30.106 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:30.106 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:30.107 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:05:30.107 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:30.107 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:30.107 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:30.295 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:30.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-nbd 2023-09-29T23:05:30.586 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:31.169 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:03:01 ago on Fri 29 Sep 2023 11:02:30 PM UTC. 2023-09-29T23:05:31.213 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:31.213 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:31.213 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-29T23:05:31.213 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:31.213 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:31.213 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-29T23:05:31.432 DEBUG:teuthology.parallel:result is None 2023-09-29T23:05:32.741 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-09-29T23:05:32.741 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:32.741 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-09-29T23:05:32.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:32.743 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 173 k 2023-09-29T23:05:32.743 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 552 k 2023-09-29T23:05:32.743 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-09-29T23:05:32.929 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 929 kB/s | 173 kB 00:00 2023-09-29T23:05:32.930 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:05:32.930 INFO:teuthology.orchestra.run.smithi123.stdout:Total 922 kB/s | 173 kB 00:00 2023-09-29T23:05:32.930 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-09-29T23:05:32.941 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-09-29T23:05:32.941 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-09-29T23:05:32.990 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-09-29T23:05:32.993 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-09-29T23:05:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-09-29T23:05:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-29T23:05:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:34.231 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-09-29T23:05:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-29T23:05:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:34.231 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-09-29T23:05:34.426 DEBUG:teuthology.parallel:result is None 2023-09-29T23:05:34.426 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:05:34.570 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-29T23:05:34.598 INFO:teuthology.orchestra.run.smithi121.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-29T23:05:34.599 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-29T23:05:34.599 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-29T23:05:34.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:05:34.817 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-29T23:05:34.841 INFO:teuthology.orchestra.run.smithi123.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-29T23:05:34.841 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-29T23:05:34.841 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-29T23:05:34.841 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-29T23:05:34.841 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:34.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T23:05:34.875 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:34.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T23:05:34.908 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-29T23:05:34.909 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:34.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T23:05:34.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T23:05:35.016 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:35.016 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T23:05:35.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T23:05:35.112 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-29T23:05:35.112 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:35.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T23:05:35.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T23:05:35.211 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:35.212 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T23:05:35.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T23:05:35.306 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-29T23:05:35.306 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:35.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T23:05:35.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T23:05:35.404 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:35.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T23:05:35.431 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T23:05:35.499 INFO:teuthology.run_tasks:Running task ceph... 2023-09-29T23:05:35.565 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-29T23:05:35.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2023-09-29T23:05:35.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2023-09-29T23:05:35.595 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-29T23:05:35.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T23:05:35.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T23:05:35.663 INFO:tasks.ceph:Creating extra log directories... 2023-09-29T23:05:35.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T23:05:35.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T23:05:35.746 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-29T23:05:35.746 INFO:tasks.ceph:config {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_DOWN\\)', '\\(OSD_HOST_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(OSDMAP_FLAGS\\)', '\\(OSD_FLAGS\\)', '\\(TELEMETRY_CHANGED\\)', 'pauserd,pausewr flag\\(s\\) set', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'evicting unresponsive client .+', 'MON_DOWN', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-29T23:05:35.747 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/dashboard}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406517', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'], ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4/SG/jST+P3fpYnvP3qqBOCdxntMhW67oJi2kYDwucLwJ6BPd99WBtmrWt5kYaiaQ0UR/qpZ7k/fy4wK9GW/g=', 'smithi123.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArobUAH9Zdgerb63Cfvmh72MtB0kienk5a9ay4DNdK59tgLV3YNl1ZfokzHBBrrcpYP12D7NyCVsOU0Gco9sTA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'pexec': {'all': ['sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup', 'sudo dnf -y module reset container-tools', 'sudo dnf -y module install container-tools --allowerasing --nobest', 'sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf', 'sudo sed -i \'s/runtime = "runc"/#runtime = "runc"/g\' /usr/share/containers/containers.conf', 'sudo sed -i \'s/#runtime = "crun"/runtime = "crun"/g\' /usr/share/containers/containers.conf']}}, {'install': None}, {'ceph': {'log-ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_DOWN\\)', '\\(OSD_HOST_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(OSDMAP_FLAGS\\)', '\\(OSD_FLAGS\\)', '\\(TELEMETRY_CHANGED\\)', 'pauserd,pausewr flag\\(s\\) set', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'evicting unresponsive client .+', 'MON_DOWN', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'wait-for-scrub': False, 'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'rgw': ['client.0']}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.mgr.test_dashboard', 'tasks.mgr.dashboard.test_api', 'tasks.mgr.dashboard.test_auth', 'tasks.mgr.dashboard.test_cephfs', 'tasks.mgr.dashboard.test_cluster', 'tasks.mgr.dashboard.test_cluster_configuration', 'tasks.mgr.dashboard.test_crush_rule', 'tasks.mgr.dashboard.test_erasure_code_profile', 'tasks.mgr.dashboard.test_health', 'tasks.mgr.dashboard.test_host', 'tasks.mgr.dashboard.test_logs', 'tasks.mgr.dashboard.test_mgr_module', 'tasks.mgr.dashboard.test_monitor', 'tasks.mgr.dashboard.test_motd', 'tasks.mgr.dashboard.test_orchestrator', 'tasks.mgr.dashboard.test_osd', 'tasks.mgr.dashboard.test_perf_counters', 'tasks.mgr.dashboard.test_pool', 'tasks.mgr.dashboard.test_rbd', 'tasks.mgr.dashboard.test_rbd_mirroring', 'tasks.mgr.dashboard.test_requests', 'tasks.mgr.dashboard.test_rgw', 'tasks.mgr.dashboard.test_role', 'tasks.mgr.dashboard.test_settings', 'tasks.mgr.dashboard.test_summary', 'tasks.mgr.dashboard.test_telemetry', 'tasks.mgr.dashboard.test_user']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-29T23:05:35.747 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-29T23:05:35.774 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-29T23:05:35.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T23:05:35.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T23:05:35.860 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:35.860 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T23:05:35.917 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T23:05:35.917 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2023-09-29T23:05:35.974 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T23:05:35.975 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:35.975 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 396359 Links: 1 2023-09-29T23:05:35.975 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:35.975 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:35.975 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-29 23:05:26.107975120 +0000 2023-09-29T23:05:35.975 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-29 23:00:59.829937089 +0000 2023-09-29T23:05:35.975 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-29 23:00:59.829937089 +0000 2023-09-29T23:05:35.976 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-29T23:05:35.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T23:05:36.040 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-29T23:05:36.041 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-29T23:05:36.041 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000223876 s, 2.3 MB/s 2023-09-29T23:05:36.042 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T23:05:36.102 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2023-09-29T23:05:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 392441 Links: 1 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-29 23:05:26.107975120 +0000 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-29 23:01:00.354930137 +0000 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-29 23:01:00.354930137 +0000 2023-09-29T23:05:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-29T23:05:36.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T23:05:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-29T23:05:36.226 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-29T23:05:36.226 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000196001 s, 2.6 MB/s 2023-09-29T23:05:36.227 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T23:05:36.285 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2023-09-29T23:05:36.344 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T23:05:36.345 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:36.345 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 383987 Links: 1 2023-09-29T23:05:36.345 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:36.346 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:36.346 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-29 23:05:26.107975120 +0000 2023-09-29T23:05:36.346 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-29 23:01:00.869923318 +0000 2023-09-29T23:05:36.346 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-29 23:01:00.869923318 +0000 2023-09-29T23:05:36.347 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-29T23:05:36.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T23:05:36.410 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-29T23:05:36.411 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-29T23:05:36.411 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00022258 s, 2.3 MB/s 2023-09-29T23:05:36.412 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T23:05:36.470 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 398492 Links: 1 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-29 23:05:26.107975120 +0000 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-29 23:01:01.423915982 +0000 2023-09-29T23:05:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-29 23:01:01.423915982 +0000 2023-09-29T23:05:36.530 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-29T23:05:36.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T23:05:36.594 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-29T23:05:36.595 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-29T23:05:36.595 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000231611 s, 2.2 MB/s 2023-09-29T23:05:36.597 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T23:05:36.654 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1'} 2023-09-29T23:05:36.655 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:36.655 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T23:05:36.676 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T23:05:36.677 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2023-09-29T23:05:36.737 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T23:05:36.737 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:36.737 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 388723 Links: 1 2023-09-29T23:05:36.737 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:36.737 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:36.737 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-09-29 23:05:29.231278034 +0000 2023-09-29T23:05:36.737 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-09-29 23:00:58.403146877 +0000 2023-09-29T23:05:36.738 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-09-29 23:00:58.403146877 +0000 2023-09-29T23:05:36.738 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-09-29T23:05:36.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T23:05:36.806 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-09-29T23:05:36.806 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-09-29T23:05:36.806 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.00026438 s, 1.9 MB/s 2023-09-29T23:05:36.807 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T23:05:36.868 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2023-09-29T23:05:36.928 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 391826 Links: 1 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-09-29 23:05:29.231278034 +0000 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-09-29 23:00:58.898135578 +0000 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-09-29 23:00:58.898135578 +0000 2023-09-29T23:05:36.929 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-09-29T23:05:36.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T23:05:36.996 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-09-29T23:05:36.997 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-09-29T23:05:36.997 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000229738 s, 2.2 MB/s 2023-09-29T23:05:36.998 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T23:05:37.057 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2023-09-29T23:05:37.116 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T23:05:37.116 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:37.116 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 393697 Links: 1 2023-09-29T23:05:37.116 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:37.116 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:37.116 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-09-29 23:05:29.231278034 +0000 2023-09-29T23:05:37.117 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-09-29 23:00:59.404124028 +0000 2023-09-29T23:05:37.117 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-09-29 23:00:59.404124028 +0000 2023-09-29T23:05:37.117 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-09-29T23:05:37.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T23:05:37.183 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-09-29T23:05:37.183 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-09-29T23:05:37.183 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000222446 s, 2.3 MB/s 2023-09-29T23:05:37.184 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T23:05:37.244 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2023-09-29T23:05:37.301 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T23:05:37.302 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T23:05:37.302 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 386025 Links: 1 2023-09-29T23:05:37.302 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T23:05:37.302 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-09-29T23:05:37.302 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-09-29 23:05:29.231278034 +0000 2023-09-29T23:05:37.302 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-09-29 23:00:59.908112524 +0000 2023-09-29T23:05:37.302 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-09-29 23:00:59.908112524 +0000 2023-09-29T23:05:37.303 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-09-29T23:05:37.303 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T23:05:37.367 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-09-29T23:05:37.368 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-09-29T23:05:37.368 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000211306 s, 2.4 MB/s 2023-09-29T23:05:37.369 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T23:05:37.427 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-29T23:05:37.428 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2023-09-29T23:05:37.428 INFO:tasks.ceph:Generating config... 2023-09-29T23:05:37.429 INFO:tasks.ceph:[client] debug client = 20 2023-09-29T23:05:37.429 INFO:tasks.ceph:[client] debug mgrc = 20 2023-09-29T23:05:37.429 INFO:tasks.ceph:[client] debug ms = 1 2023-09-29T23:05:37.429 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-29T23:05:37.429 INFO:tasks.ceph:[mds] debug mgrc = 20 2023-09-29T23:05:37.429 INFO:tasks.ceph:[mgr] debug client = 20 2023-09-29T23:05:37.430 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-29T23:05:37.430 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-29T23:05:37.430 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-29T23:05:37.430 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-29T23:05:37.430 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-29T23:05:37.430 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-29T23:05:37.430 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2023-09-29T23:05:37.430 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-29T23:05:37.430 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-29T23:05:37.430 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-09-29T23:05:37.430 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-09-29T23:05:37.430 INFO:tasks.ceph:[osd] debug mgrc = 20 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-29T23:05:37.431 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-29T23:05:37.431 INFO:tasks.ceph:Setting up mon.a... 2023-09-29T23:05:37.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-29T23:05:37.482 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.keyring 2023-09-29T23:05:37.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-29T23:05:37.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T23:05:37.559 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.121'), ('mon.c', '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]'), ('mon.b', '172.21.15.123')] 2023-09-29T23:05:37.559 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790],172.21.15.123', 'mon election default strategy': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds': {'debug mgrc': 20}} 2023-09-29T23:05:37.560 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:37.560 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T23:05:37.618 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.121 --addv c '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]' --add b 172.21.15.123 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:37.704 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: generated fsid f139d264-5213-413d-9960-69a6d67478bb 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:setting min_mon_release = octopus 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 0 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:fsid f139d264-5213-413d-9960-69a6d67478bb 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:last_changed 2023-09-29T23:05:37.705584+0000 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:created 2023-09-29T23:05:37.705584+0000 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:min_mon_release 15 (octopus) 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:election_strategy: 1 2023-09-29T23:05:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2023-09-29T23:05:37.706 INFO:teuthology.orchestra.run.smithi121.stdout:1: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2023-09-29T23:05:37.706 INFO:teuthology.orchestra.run.smithi121.stdout:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2023-09-29T23:05:37.706 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-29T23:05:37.707 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T23:05:37.725 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f139d264-5213-413d-9960-69a6d67478bb... 2023-09-29T23:05:37.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-29T23:05:37.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-29T23:05:37.810 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2023-09-29T23:05:37.810 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2023-09-29T23:05:37.810 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T23:05:37.810 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2023-09-29T23:05:37.810 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.810 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2023-09-29T23:05:37.810 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2023-09-29T23:05:37.811 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2023-09-29T23:05:37.812 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2023-09-29T23:05:37.813 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790],172.21.15.123" 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 1 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = f139d264-5213-413d-9960-69a6d67478bb 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2023-09-29T23:05:37.814 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2023-09-29T23:05:37.815 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.816 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2023-09-29T23:05:37.817 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression algorithm = zlib 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression mode = aggressive 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgrc = 20 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2023-09-29T23:05:37.818 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-09-29T23:05:37.819 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2023-09-29T23:05:37.820 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.821 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2023-09-29T23:05:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgrc = 20 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-09-29T23:05:37.823 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-09-29T23:05:37.824 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2023-09-29T23:05:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgrc = 20 2023-09-29T23:05:37.827 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2023-09-29T23:05:37.827 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2023-09-29T23:05:37.827 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T23:05:37.827 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2023-09-29T23:05:37.828 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.829 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: mon_allow_pool_size_one = true 2023-09-29T23:05:37.830 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: mon health detail to clog = false 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790],172.21.15.123" 2023-09-29T23:05:37.831 INFO:teuthology.orchestra.run.smithi123.stdout: mon election default strategy = 1 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = f139d264-5213-413d-9960-69a6d67478bb 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 600 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock profile = high_recovery_ops 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2023-09-29T23:05:37.832 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2023-09-29T23:05:37.833 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2023-09-29T23:05:37.834 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore compression algorithm = zlib 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore compression mode = aggressive 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 1/20 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 1/20 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgrc = 20 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 20 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 4/10 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2023-09-29T23:05:37.835 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock override recovery settings = True 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2023-09-29T23:05:37.836 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2023-09-29T23:05:37.837 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T23:05:37.838 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: # 1m isn't quite enough 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: mon_down_mkfs_grace = 2m 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_filestore_osds = false 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2023-09-29T23:05:37.839 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgrc = 20 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2023-09-29T23:05:37.840 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2023-09-29T23:05:37.841 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2023-09-29T23:05:37.841 INFO:teuthology.orchestra.run.smithi123.stdout:[mds] 2023-09-29T23:05:37.841 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgrc = 20 2023-09-29T23:05:37.841 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-29T23:05:37.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-29T23:05:37.921 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-29T23:05:37.922 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:37.922 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-29T23:05:37.940 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:37.941 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-29T23:05:37.999 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2023-09-29T23:05:37.999 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:37.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-29T23:05:37.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T23:05:38.075 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:38.076 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:38.133 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:05:38.134 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:38.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-29T23:05:38.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-29T23:05:38.171 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:38.172 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:38.229 INFO:tasks.ceph:Setting up mon nodes... 2023-09-29T23:05:38.230 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-29T23:05:38.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-29T23:05:38.281 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-29T23:05:38.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-29T23:05:38.349 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-29T23:05:38.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-z && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.z /var/lib/ceph/mgr/ceph-z/keyring 2023-09-29T23:05:38.402 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-z/keyring 2023-09-29T23:05:38.404 INFO:tasks.ceph:Setting up mds nodes... 2023-09-29T23:05:38.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-09-29T23:05:38.455 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-09-29T23:05:38.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-09-29T23:05:38.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-09-29T23:05:38.573 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-09-29T23:05:38.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-09-29T23:05:38.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-09-29T23:05:38.654 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-09-29T23:05:38.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-09-29T23:05:38.683 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-29T23:05:38.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-29T23:05:38.724 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-29T23:05:38.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-29T23:05:38.776 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-29T23:05:38.788 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-29T23:05:38.789 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2023-09-29T23:05:38.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-29T23:05:38.816 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1'} 2023-09-29T23:05:38.816 INFO:tasks.ceph:role: osd.0 2023-09-29T23:05:38.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-29T23:05:38.816 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T23:05:38.892 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:05:38.892 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:05:38.892 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:05:38.892 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-29T23:05:38.892 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:05:38.892 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:05:38.892 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:05:38.893 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:05:38.893 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:05:38.893 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:05:39.326 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-09-29T23:05:39.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-09-29T23:05:39.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-29T23:05:39.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-29T23:05:39.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:39.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:39.625+0000 7f421c707540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-29T23:05:39.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:39.625+0000 7f421c707540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-29T23:05:39.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:39.625+0000 7f421c707540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-29T23:05:39.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:39.625+0000 7f421c707540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-29T23:05:39.627 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:39.625+0000 7f421c707540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-29T23:05:39.631 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:39.631+0000 7f421c707540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-29T23:05:40.158 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:40.158+0000 7f421c707540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:40.924 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:40.923+0000 7f421c707540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:41.690 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:41.689+0000 7f421c707540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:42.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-29T23:05:42.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-29T23:05:42.551 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-29T23:05:42.551 INFO:tasks.ceph:role: osd.1 2023-09-29T23:05:42.551 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:05:42.551 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T23:05:42.631 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:05:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:05:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:05:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-29T23:05:42.632 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:05:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:05:42.632 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:05:42.632 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:05:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:05:42.632 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:05:43.062 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-09-29T23:05:43.064 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-09-29T23:05:43.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2023-09-29T23:05:43.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-29T23:05:43.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-29T23:05:43.363 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-29T23:05:43.363 INFO:tasks.ceph:role: osd.2 2023-09-29T23:05:43.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:05:43.363 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T23:05:43.438 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:05:43.439 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:05:43.887 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-09-29T23:05:43.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-09-29T23:05:43.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2023-09-29T23:05:44.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-29T23:05:44.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-29T23:05:44.105 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-29T23:05:44.105 INFO:tasks.ceph:role: osd.3 2023-09-29T23:05:44.105 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:05:44.105 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T23:05:44.181 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T23:05:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T23:05:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T23:05:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-29T23:05:44.181 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T23:05:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-09-29T23:05:44.181 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T23:05:44.182 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T23:05:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T23:05:44.182 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T23:05:44.635 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-09-29T23:05:44.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-09-29T23:05:44.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2023-09-29T23:05:44.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-29T23:05:44.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:44.928 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:44.927+0000 7f0b85cf4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-29T23:05:44.930 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:44.930+0000 7f0b85cf4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-29T23:05:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:44.930+0000 7f0b85cf4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-29T23:05:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:44.930+0000 7f0b85cf4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-29T23:05:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:44.930+0000 7f0b85cf4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-29T23:05:44.932 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:44.931+0000 7f0b85cf4540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-29T23:05:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:45.456+0000 7f0b85cf4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:46.221 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:46.220+0000 7f0b85cf4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:46.984 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:46.983+0000 7f0b85cf4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:47.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-29T23:05:47.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:47.915+0000 7f4ecca25540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-29T23:05:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:47.915+0000 7f4ecca25540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-29T23:05:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:47.915+0000 7f4ecca25540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-29T23:05:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:47.915+0000 7f4ecca25540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-29T23:05:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:47.915+0000 7f4ecca25540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-29T23:05:47.917 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:47.915+0000 7f4ecca25540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-29T23:05:48.443 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:48.442+0000 7f4ecca25540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:49.209+0000 7f4ecca25540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:49.972 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:49.971+0000 7f4ecca25540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:50.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-29T23:05:50.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:50.895 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:50.894+0000 7fe317ffa540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-29T23:05:50.896 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:50.894+0000 7fe317ffa540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-29T23:05:50.896 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:50.894+0000 7fe317ffa540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-29T23:05:50.896 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:50.894+0000 7fe317ffa540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-29T23:05:50.896 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:50.894+0000 7fe317ffa540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-29T23:05:50.896 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:50.895+0000 7fe317ffa540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-29T23:05:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:51.419+0000 7fe317ffa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:52.182 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:52.181+0000 7fe317ffa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:52.944+0000 7fe317ffa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-29T23:05:53.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-29T23:05:53.774 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-29T23:05:53.774 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:53.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-29T23:05:53.807 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:53.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-09-29T23:05:53.877 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:53.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-09-29T23:05:53.947 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:53.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-29T23:05:54.016 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:54.016 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-29T23:05:54.050 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:54.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-z/keyring of=/dev/stdout 2023-09-29T23:05:54.123 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:54.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-09-29T23:05:54.196 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:54.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-29T23:05:54.269 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:54.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-29T23:05:54.342 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:54.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-29T23:05:54.412 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:54.412 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-29T23:05:54.432 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:54.432 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-29T23:05:54.471 INFO:tasks.ceph:Adding keys to all mons... 2023-09-29T23:05:54.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-29T23:05:54.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-29T23:05:54.514 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2023-09-29T23:05:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdlobbIEBAAhk5MNvEFYrQrz//m0WUM1A== 2023-09-29T23:05:54.514 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-09-29T23:05:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdlMIksGxAAnFJijQP8DQmkrrXWju84Cg== 2023-09-29T23:05:54.514 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdlzG8/IhAAhFiPPeBY7Xxp7bvRMLOnBA== 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBDWBdlLDNcJRAAxd+QWz4cqHYgidtK+MMhvQ== 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdlHrfaFBAAzXyhLsRbmBNd8IU6WuNIlQ== 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.z] 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdl+tkDGBAAiryOIxaTWs8z2q21+uUjbw== 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdl6l8OJxAAoZZGnQ5z5uWEkpnnVQIVcg== 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBIWBdlG0lnNxAAziIvdKAwgDyKw3ppoQ9rjw== 2023-09-29T23:05:54.515 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2023-09-29T23:05:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBLWBdlATakNhAAHdPonOTyD9N69H1UizZpCA== 2023-09-29T23:05:54.516 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2023-09-29T23:05:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBOWBdl5RVqNRAAAMengyD385+w07XVSf4/vQ== 2023-09-29T23:05:54.516 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2023-09-29T23:05:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdlMiU/KxAAKTnS+Gmwqnnn9aE4EJtZIw== 2023-09-29T23:05:54.516 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2023-09-29T23:05:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBCWBdlobVZLhAAnfkSI4CREl9C9h6LeeyL/w== 2023-09-29T23:05:54.538 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2023-09-29T23:05:54.538 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdlobbIEBAAhk5MNvEFYrQrz//m0WUM1A== 2023-09-29T23:05:54.538 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2023-09-29T23:05:54.538 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdlMIksGxAAnFJijQP8DQmkrrXWju84Cg== 2023-09-29T23:05:54.538 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2023-09-29T23:05:54.538 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdlzG8/IhAAhFiPPeBY7Xxp7bvRMLOnBA== 2023-09-29T23:05:54.538 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBDWBdlLDNcJRAAxd+QWz4cqHYgidtK+MMhvQ== 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdlHrfaFBAAzXyhLsRbmBNd8IU6WuNIlQ== 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.z] 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdl+tkDGBAAiryOIxaTWs8z2q21+uUjbw== 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdl6l8OJxAAoZZGnQ5z5uWEkpnnVQIVcg== 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBIWBdlG0lnNxAAziIvdKAwgDyKw3ppoQ9rjw== 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBLWBdlATakNhAAHdPonOTyD9N69H1UizZpCA== 2023-09-29T23:05:54.539 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2023-09-29T23:05:54.540 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBOWBdl5RVqNRAAAMengyD385+w07XVSf4/vQ== 2023-09-29T23:05:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2023-09-29T23:05:54.540 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdlMiU/KxAAKTnS+Gmwqnnn9aE4EJtZIw== 2023-09-29T23:05:54.540 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2023-09-29T23:05:54.540 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCWBdlobVZLhAAnfkSI4CREl9C9h6LeeyL/w== 2023-09-29T23:05:54.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:05:54.560 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:05:54.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-29T23:05:54.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-29T23:05:54.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-29T23:05:54.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-29T23:05:54.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:54.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:54.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:05:54.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:05:54.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:05:54.848 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T23:05:54.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-29T23:05:54.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-29T23:05:54.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:54.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:55.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:55.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:55.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:55.060 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T23:05:55.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T23:05:55.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T23:05:55.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T23:05:55.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T23:05:55.219 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-29T23:05:55.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-29T23:05:55.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:05:55.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-29T23:05:55.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-29T23:05:55.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:05:55.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-29T23:05:55.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-29T23:05:55.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-29T23:05:55.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-29T23:05:55.708 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:55.710 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-29T23:05:55.766 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-29T23:05:55.766 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-29T23:05:55.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-29T23:05:55.768 INFO:tasks.ceph.mon.a:Started 2023-09-29T23:05:55.769 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-29T23:05:55.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-29T23:05:55.770 INFO:tasks.ceph.mon.c:Started 2023-09-29T23:05:55.771 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-29T23:05:55.771 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-29T23:05:55.808 INFO:tasks.ceph.mon.b:Started 2023-09-29T23:05:55.808 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-29T23:05:55.808 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-29T23:05:55.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-29T23:05:55.810 INFO:tasks.ceph.mgr.x:Started 2023-09-29T23:05:55.811 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-29T23:05:55.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-29T23:05:55.812 INFO:tasks.ceph.mgr.y:Started 2023-09-29T23:05:55.813 INFO:tasks.ceph.mgr.z:Restarting daemon 2023-09-29T23:05:55.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i z 2023-09-29T23:05:55.814 INFO:tasks.ceph.mgr.z:Started 2023-09-29T23:05:55.815 INFO:tasks.ceph:Setting crush tunables to default 2023-09-29T23:05:55.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd crush tunables default 2023-09-29T23:05:55.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.935+0000 7f20588d8700 1 Processor -- start 2023-09-29T23:05:55.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.935+0000 7f20588d8700 1 -- start start 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f20588d8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f20541344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f20588d8700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 0x7f20540b4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f20588d8700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20540b3990 con 0x7f20541349f0 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f20588d8700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205406e000 con 0x7f20540b3e20 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f20588d8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20540b47b0 con 0x7f205409e4e0 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f2052d9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 0x7f20540b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f205259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f20541344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:55.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f2052d9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 0x7f20540b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43634/0 (socket says 172.21.15.121:43634) 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f205259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f20541344b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:55004/0 (socket says 172.21.15.121:55004) 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f2052d9d700 1 -- 172.21.15.121:0/4253396215 learned_addr learned my addr 172.21.15.121:0/4253396215 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f2052d9d700 1 -- 172.21.15.121:0/4253396215 >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 msgr2=0x7f20540b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f2052d9d700 1 -- 172.21.15.121:0/4253396215 >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 msgr2=0x7f20540b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f2052d9d700 1 --2- 172.21.15.121:0/4253396215 >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 0x7f20540b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f205259c700 1 -- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 msgr2=0x7f20541344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f205259c700 1 -- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 msgr2=0x7f20541344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f205259c700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f20541344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f2052d9d700 1 --2- 172.21.15.121:0/4253396215 >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 0x7f20540b4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:05:55.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:55.936+0000 7f205259c700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f20541344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-29T23:05:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.106+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589645489 0 0) 0x7f20540b3990 con 0x7f20541349f0 2023-09-29T23:05:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.106+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c003570 con 0x7f20541349f0 2023-09-29T23:05:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2665289049 0 0) 0x7f203c003570 con 0x7f20541349f0 2023-09-29T23:05:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 msgr2=0x7f20541344b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:05:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f20541344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 msgr2=0x7f20540b4270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:05:56.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 --2- 172.21.15.121:0/4253396215 >> v2:172.21.15.121:3300/0 conn(0x7f20540b3e20 0x7f20540b4270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205409e8b0 con 0x7f20541349f0 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7f2044001470 con 0x7f20541349f0 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 >> v1:172.21.15.123:6789/0 conn(0x7f20541349f0 legacy=0x7f20540b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f203c003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c004090 0x7f203c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f203c004d50 0x7f203c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20540b47b0 con 0x7f203c004d50 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c00bb70 con 0x7f203c004090 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c00bd30 con 0x7f205409e4e0 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205259c700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f203c003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f2052d9d700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f203c004d50 0x7f203c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f2051d9b700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c004090 0x7f203c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205259c700 1 -- 172.21.15.121:0/4253396215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c004090 msgr2=0x7f203c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205259c700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c004090 0x7f203c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205259c700 1 -- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f203c004d50 msgr2=0x7f203c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205259c700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f203c004d50 0x7f203c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205259c700 1 -- 172.21.15.121:0/4253396215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c003570 con 0x7f205409e4e0 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f2052d9d700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f203c004d50 0x7f203c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.107+0000 7f205259c700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f203c003a60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f204800bb40 tx=0x7f2048008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9388ae9896dce475 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.108+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2048018070 con 0x7f205409e4e0 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.108+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204800d5d0 con 0x7f205409e4e0 2023-09-29T23:05:56.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.108+0000 7f205159a700 1 -- 172.21.15.121:0/4253396215 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204800d970 con 0x7f205409e4e0 2023-09-29T23:05:56.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.108+0000 7f20588d8700 1 -- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 msgr2=0x7f203c003a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.108+0000 7f20588d8700 1 --2- 172.21.15.121:0/4253396215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205409e4e0 0x7f203c003a60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f204800bb40 tx=0x7f2048008040 comp rx=0 tx=0).stop 2023-09-29T23:05:56.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.109+0000 7f2052d9d700 1 -- 172.21.15.121:0/4253396215 reap_dead start 2023-09-29T23:05:56.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.109+0000 7f20588d8700 1 -- 172.21.15.121:0/4253396215 shutdown_connections 2023-09-29T23:05:56.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.109+0000 7f20588d8700 1 -- 172.21.15.121:0/4253396215 >> 172.21.15.121:0/4253396215 conn(0x7f2054067440 msgr2=0x7f2054067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:56.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.109+0000 7f20588d8700 1 -- 172.21.15.121:0/4253396215 shutdown_connections 2023-09-29T23:05:56.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.109+0000 7f20588d8700 1 -- 172.21.15.121:0/4253396215 wait complete. 2023-09-29T23:05:56.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.110+0000 7f20588d8700 1 Processor -- start 2023-09-29T23:05:56.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.110+0000 7f20588d8700 1 -- start start 2023-09-29T23:05:56.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f20588d8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f205409e4e0 0x7f20540a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f20588d8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2054095af0 0x7f2054095f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f20588d8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205413f4b0 0x7f205413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f20588d8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20540a4210 con 0x7f205413f4b0 2023-09-29T23:05:56.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f20588d8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2054099b00 con 0x7f205409e4e0 2023-09-29T23:05:56.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f20588d8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2054096780 con 0x7f2054095af0 2023-09-29T23:05:56.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f205259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f205409e4e0 0x7f20540a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f2051d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2054095af0 0x7f2054095f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f2052d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205413f4b0 0x7f205413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f2051d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2054095af0 0x7f2054095f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:55042/0 (socket says 172.21.15.121:55042) 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.111+0000 7f2051d9b700 1 -- 172.21.15.121:0/3914409698 learned_addr learned my addr 172.21.15.121:0/3914409698 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f2051d9b700 1 -- 172.21.15.121:0/3914409698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f205409e4e0 msgr2=0x7f20540a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f2051d9b700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f205409e4e0 0x7f20540a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f2051d9b700 1 -- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205413f4b0 msgr2=0x7f205413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f2051d9b700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205413f4b0 0x7f205413f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f2051d9b700 1 -- 172.21.15.121:0/3914409698 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2054096a40 con 0x7f2054095af0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f2052d9d700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205413f4b0 0x7f205413f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f2051d9b700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2054095af0 0x7f2054095f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2044010f50 tx=0x7f2044012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.112+0000 7f20427fc700 1 -- 172.21.15.121:0/3914409698 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044015040 con 0x7f2054095af0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 1 -- 172.21.15.121:0/3914409698 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2044013b30 con 0x7f2054095af0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2054098a80 con 0x7f2054095af0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 1 -- 172.21.15.121:0/3914409698 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044013ed0 con 0x7f2054095af0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20540a3b70 con 0x7f2054095af0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 1 -- 172.21.15.121:0/3914409698 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f204401b600 con 0x7f2054095af0 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 4 mgrc handle_mgr_map Got map version 1 2023-09-29T23:05:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 4 mgrc handle_mgr_map Active mgr is now 2023-09-29T23:05:56.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 4 mgrc reconnect No active mgr available yet 2023-09-29T23:05:56.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.113+0000 7f20427fc700 1 -- 172.21.15.121:0/3914409698 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2044025860 con 0x7f2054095af0 2023-09-29T23:05:56.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.114+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2054099c70 con 0x7f2054095af0 2023-09-29T23:05:56.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.116+0000 7f20427fc700 1 -- 172.21.15.121:0/3914409698 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2044020070 con 0x7f2054095af0 2023-09-29T23:05:56.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.195+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2054071660 con 0x7f2054095af0 2023-09-29T23:05:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.199+0000 7f20427fc700 1 -- 172.21.15.121:0/3914409698 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f204401bae0 con 0x7f2054095af0 2023-09-29T23:05:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.199+0000 7f20588d8700 10 mgrc shutdown 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2054095af0 msgr2=0x7f2054095f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2054095af0 0x7f2054095f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2044010f50 tx=0x7f2044012040 comp rx=0 tx=0).stop 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 shutdown_connections 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205413f4b0 0x7f205413f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f205409e4e0 0x7f20540a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 --2- 172.21.15.121:0/3914409698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2054095af0 0x7f2054095f20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 >> 172.21.15.121:0/3914409698 conn(0x7f2054067440 msgr2=0x7f20540a4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 shutdown_connections 2023-09-29T23:05:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:56.200+0000 7f20588d8700 1 -- 172.21.15.121:0/3914409698 wait complete. 2023-09-29T23:05:56.202 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2023-09-29T23:05:56.212 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-29T23:05:56.212 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-29T23:05:56.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-29T23:05:56.221 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:56.221+0000 7f2e3246c1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T23:05:56.239 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:56.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-29T23:05:56.247 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:56.247+0000 7f4c5ac5f1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T23:05:56.266 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:56.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-29T23:05:56.345 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-09-29T23:05:56.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-29T23:05:56.403 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:56.402+0000 7f2e3246c1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T23:05:56.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 668222c3-dafb-41e3-9a19-feeeafa50f67 0 2023-09-29T23:05:56.412 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:56.412+0000 7f4c5ac5f1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T23:05:56.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.572+0000 7f9994d59700 1 Processor -- start 2023-09-29T23:05:56.572 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.572+0000 7f9994d59700 1 -- start start 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f9994d59700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f999006b080 0x7f99900b4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f9994d59700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 0x7f9990143a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f9994d59700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900b4870 con 0x7f999006acb0 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f9994d59700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9990143fc0 con 0x7f999006b080 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f9994d59700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b4b30 con 0x7f99900b3a90 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998e59c700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f999006acb0 0x7f9990144400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:60632/0 (socket says 172.21.15.123:60632) 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998e59c700 1 -- 172.21.15.123:0/27589647 learned_addr learned my addr 172.21.15.123:0/27589647 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-09-29T23:05:56.576 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998ed9d700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 0x7f9990143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472128524 0 0) 0x7f99900b4870 con 0x7f999006acb0 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9978003570 con 0x7f999006acb0 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998dd9b700 1 --2- 172.21.15.123:0/27589647 >> v2:172.21.15.121:3300/0 conn(0x7f999006b080 0x7f99900b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 897648853 0 0) 0x7f9978003570 con 0x7f999006acb0 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 msgr2=0x7f9990143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 0x7f9990143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 >> v2:172.21.15.121:3300/0 conn(0x7f999006b080 msgr2=0x7f99900b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 --2- 172.21.15.123:0/27589647 >> v2:172.21.15.121:3300/0 conn(0x7f999006b080 0x7f99900b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990069be0 con 0x7f999006acb0 2023-09-29T23:05:56.577 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7f99800015f0 con 0x7f999006acb0 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 >> v1:172.21.15.123:6789/0 conn(0x7f999006acb0 legacy=0x7f9990144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9978003a60 0x7f9978003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978004810 0x7f9978004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 0x7f99780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990143fc0 con 0x7f9978004810 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99900b4b30 con 0x7f9978003a60 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.573+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997800ba60 con 0x7f99900b3a90 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998e59c700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9978003a60 0x7f9978003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998dd9b700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978004810 0x7f9978004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.578 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998ed9d700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 0x7f99780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998e59c700 1 -- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 msgr2=0x7f99780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998e59c700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99900b3a90 0x7f99780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998e59c700 1 -- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978004810 msgr2=0x7f9978004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998e59c700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978004810 0x7f9978004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998e59c700 1 -- 172.21.15.123:0/27589647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978003570 con 0x7f9978003a60 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998e59c700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9978003a60 0x7f9978003e70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f998000c3c0 tx=0x7f9980007920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68f6b54716156f6f server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998000cc90 con 0x7f9978003a60 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998000cdf0 con 0x7f9978003a60 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.574+0000 7f998d59a700 1 -- 172.21.15.123:0/27589647 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9980007320 con 0x7f9978003a60 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.575+0000 7f9994d59700 1 -- 172.21.15.123:0/27589647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9978003a60 msgr2=0x7f9978003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.579 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.575+0000 7f9994d59700 1 --2- 172.21.15.123:0/27589647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9978003a60 0x7f9978003e70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f998000c3c0 tx=0x7f9980007920 comp rx=0 tx=0).stop 2023-09-29T23:05:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.575+0000 7f998ed9d700 1 -- 172.21.15.123:0/27589647 reap_dead start 2023-09-29T23:05:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.575+0000 7f9994d59700 1 -- 172.21.15.123:0/27589647 shutdown_connections 2023-09-29T23:05:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.575+0000 7f9994d59700 1 -- 172.21.15.123:0/27589647 >> 172.21.15.123:0/27589647 conn(0x7f999006d160 msgr2=0x7f999006d560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.575+0000 7f9994d59700 1 -- 172.21.15.123:0/27589647 shutdown_connections 2023-09-29T23:05:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.575+0000 7f9994d59700 1 -- 172.21.15.123:0/27589647 wait complete. 2023-09-29T23:05:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.576+0000 7f9994d59700 1 Processor -- start 2023-09-29T23:05:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.576+0000 7f9994d59700 1 -- start start 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f9994d59700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9978003a60 0x7f9990142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f9994d59700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9990150280 0x7f9990150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f9994d59700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99900a0260 0x7f99900a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f9994d59700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990151500 con 0x7f99900a0260 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f9994d59700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99900a8be0 con 0x7f9990150280 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f9994d59700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9990150f20 con 0x7f9978003a60 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9990150280 0x7f9990150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9990150280 0x7f9990150650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56386/0 (socket says 172.21.15.123:56386) 2023-09-29T23:05:56.583 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 -- 172.21.15.123:0/1142907383 learned_addr learned my addr 172.21.15.123:0/1142907383 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998e59c700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9978003a60 0x7f9990142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998ed9d700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99900a0260 0x7f99900a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 -- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9978003a60 msgr2=0x7f9990142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9978003a60 0x7f9990142e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 -- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99900a0260 msgr2=0x7f99900a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99900a0260 0x7f99900a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 -- 172.21.15.123:0/1142907383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99901511e0 con 0x7f9990150280 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.577+0000 7f998dd9b700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9990150280 0x7f9990150650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9984011d70 tx=0x7f99840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 1 -- 172.21.15.123:0/1142907383 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998400fa20 con 0x7f9990150280 2023-09-29T23:05:56.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 1 -- 172.21.15.123:0/1142907383 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998400fb80 con 0x7f9990150280 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 1 -- 172.21.15.123:0/1142907383 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984014440 con 0x7f9990150280 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99900a8900 con 0x7f9990150280 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999009e580 con 0x7f9990150280 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 1 -- 172.21.15.123:0/1142907383 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9984014af0 con 0x7f9990150280 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 4 mgrc handle_mgr_map Got map version 1 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 4 mgrc handle_mgr_map Active mgr is now 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 4 mgrc reconnect No active mgr available yet 2023-09-29T23:05:56.585 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.578+0000 7f997e7fc700 1 -- 172.21.15.123:0/1142907383 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9984018030 con 0x7f9990150280 2023-09-29T23:05:56.586 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.579+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9990071660 con 0x7f9990150280 2023-09-29T23:05:56.586 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.581+0000 7f997e7fc700 1 -- 172.21.15.123:0/1142907383 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9984022060 con 0x7f9990150280 2023-09-29T23:05:56.601 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:56.601+0000 7fb0d7a7b1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T23:05:56.664 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.664+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "668222c3-dafb-41e3-9a19-feeeafa50f67", "id": 0} v 0) v1 -- 0x7f99900b4050 con 0x7f9990150280 2023-09-29T23:05:56.669 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.669+0000 7f997e7fc700 1 -- 172.21.15.123:0/1142907383 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "668222c3-dafb-41e3-9a19-feeeafa50f67", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f998402c0b0 con 0x7f9990150280 2023-09-29T23:05:56.669 INFO:teuthology.orchestra.run.smithi123.stdout:0 2023-09-29T23:05:56.670 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.670+0000 7f9994d59700 10 mgrc shutdown 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9990150280 msgr2=0x7f9990150650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9990150280 0x7f9990150650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9984011d70 tx=0x7f99840091a0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 shutdown_connections 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99900a0260 0x7f99900a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9978003a60 0x7f9990142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 --2- 172.21.15.123:0/1142907383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9990150280 0x7f9990150650 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 >> 172.21.15.123:0/1142907383 conn(0x7f999006d160 msgr2=0x7f9990070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:56.671 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 shutdown_connections 2023-09-29T23:05:56.672 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.671+0000 7f9994d59700 1 -- 172.21.15.123:0/1142907383 wait complete. 2023-09-29T23:05:56.683 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new eb400e3d-c1be-4433-99df-ccbdabfa1054 1 2023-09-29T23:05:56.750 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:56.746+0000 7fb0d7a7b1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T23:05:56.759 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:56.759+0000 7f4c5ac5f1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T23:05:56.804 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:56.804+0000 7f2e3246c1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T23:05:56.844 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.844+0000 7fcb591b0700 1 Processor -- start 2023-09-29T23:05:56.844 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.844+0000 7fcb591b0700 1 -- start start 2023-09-29T23:05:56.846 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb591b0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb5409f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb591b0700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5409f760 con 0x7fcb540634b0 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb591b0700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5409faf0 con 0x7fcb540a2a00 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb591b0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5409d570 con 0x7fcb540630e0 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb5259c700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fcb540a2a00 0x7fcb540b38f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:60634/0 (socket says 172.21.15.123:60634) 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb5259c700 1 -- 172.21.15.123:0/3473019059 learned_addr learned my addr 172.21.15.123:0/3473019059 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb52d9d700 1 --2- 172.21.15.123:0/3473019059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb5409f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb51d9b700 1 -- 172.21.15.123:0/3473019059 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156117835 0 0) 0x7fcb5409faf0 con 0x7fcb540a2a00 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb51d9b700 1 -- 172.21.15.123:0/3473019059 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb3c003570 con 0x7fcb540a2a00 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb51d9b700 1 -- 172.21.15.123:0/3473019059 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033212333 0 0) 0x7fcb4c001490 con 0x7fcb540634b0 2023-09-29T23:05:56.847 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb51d9b700 1 -- 172.21.15.123:0/3473019059 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb5409faf0 con 0x7fcb540634b0 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb52d9d700 1 -- 172.21.15.123:0/3473019059 >> v1:172.21.15.123:6789/0 conn(0x7fcb540a2a00 legacy=0x7fcb540b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb52d9d700 1 -- 172.21.15.123:0/3473019059 >> v1:172.21.15.121:6789/0 conn(0x7fcb540634b0 legacy=0x7fcb541478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.845+0000 7fcb52d9d700 1 -- 172.21.15.123:0/3473019059 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5409d810 con 0x7fcb540630e0 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb52d9d700 1 --2- 172.21.15.123:0/3473019059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb5409f220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009910 tx=0x7fcb440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42d2ace01f5de3e4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb51d9b700 1 -- 172.21.15.123:0/3473019059 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44011630 con 0x7fcb540630e0 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb51d9b700 1 -- 172.21.15.123:0/3473019059 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb44011790 con 0x7fcb540630e0 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb51d9b700 1 -- 172.21.15.123:0/3473019059 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44011b30 con 0x7fcb540630e0 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb591b0700 1 -- 172.21.15.123:0/3473019059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 msgr2=0x7fcb5409f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb591b0700 1 --2- 172.21.15.123:0/3473019059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb5409f220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009910 tx=0x7fcb440091a0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb591b0700 1 -- 172.21.15.123:0/3473019059 shutdown_connections 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb591b0700 1 --2- 172.21.15.123:0/3473019059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb5409f220 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.848 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb591b0700 1 -- 172.21.15.123:0/3473019059 >> 172.21.15.123:0/3473019059 conn(0x7fcb54069ec0 msgr2=0x7fcb540b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:56.849 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb591b0700 1 -- 172.21.15.123:0/3473019059 shutdown_connections 2023-09-29T23:05:56.849 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.846+0000 7fcb591b0700 1 -- 172.21.15.123:0/3473019059 wait complete. 2023-09-29T23:05:56.851 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.848+0000 7fcb591b0700 1 Processor -- start 2023-09-29T23:05:56.851 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.848+0000 7fcb591b0700 1 -- start start 2023-09-29T23:05:56.851 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb591b0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb540a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.851 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb591b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb540634b0 0x7fcb54095a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.851 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb591b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb54098100 0x7fcb541367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:56.851 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb591b0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb54096390 con 0x7fcb540634b0 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb591b0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb54096650 con 0x7fcb54098100 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb591b0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb540a4dc0 con 0x7fcb540630e0 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb540634b0 0x7fcb54095a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb52d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb540a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb5359e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb54098100 0x7fcb541367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb540634b0 0x7fcb54095a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.123:34860/0 (socket says 172.21.15.123:34860) 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb5259c700 1 -- 172.21.15.123:0/1714486848 learned_addr learned my addr 172.21.15.123:0/1714486848 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb52d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb540a4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.123:54430/0 (socket says 172.21.15.123:54430) 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb5259c700 1 -- 172.21.15.123:0/1714486848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 msgr2=0x7fcb540a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb5259c700 1 --2- 172.21.15.123:0/1714486848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb540a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.852 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.849+0000 7fcb5259c700 1 -- 172.21.15.123:0/1714486848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb54098100 msgr2=0x7fcb541367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb5259c700 1 --2- 172.21.15.123:0/1714486848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb54098100 0x7fcb541367f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb5259c700 1 -- 172.21.15.123:0/1714486848 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44009420 con 0x7fcb540634b0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb5259c700 1 --2- 172.21.15.123:0/1714486848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb540634b0 0x7fcb54095a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c790 tx=0x7fcb48007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 1 -- 172.21.15.123:0/1714486848 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48005b30 con 0x7fcb540634b0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 1 -- 172.21.15.123:0/1714486848 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb48005cd0 con 0x7fcb540634b0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5406f590 con 0x7fcb540634b0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 1 -- 172.21.15.123:0/1714486848 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48007320 con 0x7fcb540634b0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5406e8c0 con 0x7fcb540634b0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 1 -- 172.21.15.123:0/1714486848 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcb4801c6c0 con 0x7fcb540634b0 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 4 mgrc handle_mgr_map Got map version 1 2023-09-29T23:05:56.853 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 4 mgrc handle_mgr_map Active mgr is now 2023-09-29T23:05:56.854 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 4 mgrc reconnect No active mgr available yet 2023-09-29T23:05:56.854 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.850+0000 7fcb3affd700 1 -- 172.21.15.123:0/1714486848 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fcb480127c0 con 0x7fcb540634b0 2023-09-29T23:05:56.854 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.851+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5406ab80 con 0x7fcb540634b0 2023-09-29T23:05:56.854 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.853+0000 7fcb3affd700 1 -- 172.21.15.123:0/1714486848 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fcb48021060 con 0x7fcb540634b0 2023-09-29T23:05:56.970 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.971+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "eb400e3d-c1be-4433-99df-ccbdabfa1054", "id": 1} v 0) v1 -- 0x7fcb5406ea50 con 0x7fcb540634b0 2023-09-29T23:05:56.975 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.976+0000 7fcb3affd700 1 -- 172.21.15.123:0/1714486848 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb400e3d-c1be-4433-99df-ccbdabfa1054", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcb4802a0b0 con 0x7fcb540634b0 2023-09-29T23:05:56.975 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-09-29T23:05:56.977 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 10 mgrc shutdown 2023-09-29T23:05:56.977 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb540634b0 msgr2=0x7fcb54095a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:56.977 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 --2- 172.21.15.123:0/1714486848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb540634b0 0x7fcb54095a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c790 tx=0x7fcb48007e80 comp rx=0 tx=0).stop 2023-09-29T23:05:56.977 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 shutdown_connections 2023-09-29T23:05:56.978 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 --2- 172.21.15.123:0/1714486848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb54098100 0x7fcb541367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.978 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 --2- 172.21.15.123:0/1714486848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb540630e0 0x7fcb540a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.978 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 --2- 172.21.15.123:0/1714486848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb540634b0 0x7fcb54095a10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:56.978 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 >> 172.21.15.123:0/1714486848 conn(0x7fcb54069ec0 msgr2=0x7fcb540b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:56.978 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 shutdown_connections 2023-09-29T23:05:56.978 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:56.977+0000 7fcb591b0700 1 -- 172.21.15.123:0/1714486848 wait complete. 2023-09-29T23:05:56.992 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 8304c4c7-39e6-4a99-bcf5-d39fb111241c 2 2023-09-29T23:05:57.029 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:57.029+0000 7fb0d7a7b1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T23:05:57.113 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.113+0000 7f10e6912700 1 Processor -- start 2023-09-29T23:05:57.114 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.113+0000 7f10e6912700 1 -- start start 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10e6912700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f10e00630e0 0x7f10e00b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10e6912700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00634b0 0x7f10e009bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10e6912700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e00a03c0 con 0x7f10e00a4570 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10e6912700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e009c940 con 0x7f10e00630e0 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10e6912700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e009fac0 con 0x7f10e00634b0 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10dffff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f10e00630e0 0x7f10e00b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10dffff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f10e00630e0 0x7f10e00b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56408/0 (socket says 172.21.15.123:56408) 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10dffff700 1 -- 172.21.15.123:0/2416832225 learned_addr learned my addr 172.21.15.123:0/2416832225 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-09-29T23:05:57.116 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10df7fe700 1 --2- 172.21.15.123:0/2416832225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00634b0 0x7f10e009bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10dffff700 1 -- 172.21.15.123:0/2416832225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00634b0 msgr2=0x7f10e009bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10dffff700 1 --2- 172.21.15.123:0/2416832225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00634b0 0x7f10e009bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10dffff700 1 -- 172.21.15.123:0/2416832225 >> v1:172.21.15.121:6789/0 conn(0x7f10e00a4570 legacy=0x7f10e00a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10dffff700 1 -- 172.21.15.123:0/2416832225 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00a5c80 con 0x7f10e00630e0 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.114+0000 7f10df7fe700 1 --2- 172.21.15.123:0/2416832225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00634b0 0x7f10e009bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.115+0000 7f10dffff700 1 --2- 172.21.15.123:0/2416832225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00630e0 0x7f10e00b38f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10d00099b0 tx=0x7f10d0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd38308d131992c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.115+0000 7f10deffd700 1 -- 172.21.15.123:0/2416832225 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d000b860 con 0x7f10e00630e0 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.115+0000 7f10deffd700 1 -- 172.21.15.123:0/2416832225 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d000b9c0 con 0x7f10e00630e0 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.115+0000 7f10deffd700 1 -- 172.21.15.123:0/2416832225 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d000bd60 con 0x7f10e00630e0 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.115+0000 7f10e6912700 1 -- 172.21.15.123:0/2416832225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00630e0 msgr2=0x7f10e00b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.115+0000 7f10e6912700 1 --2- 172.21.15.123:0/2416832225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00630e0 0x7f10e00b38f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10d00099b0 tx=0x7f10d0009250 comp rx=0 tx=0).stop 2023-09-29T23:05:57.118 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.116+0000 7f10e6912700 1 -- 172.21.15.123:0/2416832225 shutdown_connections 2023-09-29T23:05:57.118 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.116+0000 7f10e6912700 1 --2- 172.21.15.123:0/2416832225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00630e0 0x7f10e00b38f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.118 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.116+0000 7f10e6912700 1 --2- 172.21.15.123:0/2416832225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00634b0 0x7f10e009bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.118 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.116+0000 7f10e6912700 1 -- 172.21.15.123:0/2416832225 >> 172.21.15.123:0/2416832225 conn(0x7f10e0069ec0 msgr2=0x7f10e00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:57.118 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.116+0000 7f10e6912700 1 -- 172.21.15.123:0/2416832225 shutdown_connections 2023-09-29T23:05:57.118 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.116+0000 7f10e6912700 1 -- 172.21.15.123:0/2416832225 wait complete. 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.117+0000 7f10e6912700 1 Processor -- start 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.117+0000 7f10e6912700 1 -- start start 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10e6912700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00630e0 0x7f10e009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10e6912700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10e00634b0 0x7f10e006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10e6912700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00a4570 0x7f10e013ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10e6912700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e009e4a0 con 0x7f10e00634b0 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10e6912700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e006e2b0 con 0x7f10e00a4570 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10e6912700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e009d7d0 con 0x7f10e00630e0 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10e00634b0 0x7f10e006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.121 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10dffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00630e0 0x7f10e009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10e4eaf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00a4570 0x7f10e013ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10e00634b0 0x7f10e006eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.123:34864/0 (socket says 172.21.15.123:34864) 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 -- 172.21.15.123:0/1183807655 learned_addr learned my addr 172.21.15.123:0/1183807655 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 -- 172.21.15.123:0/1183807655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00630e0 msgr2=0x7f10e009cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 --2- 172.21.15.123:0/1183807655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00630e0 0x7f10e009cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 -- 172.21.15.123:0/1183807655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00a4570 msgr2=0x7f10e013ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 --2- 172.21.15.123:0/1183807655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00a4570 0x7f10e013ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.118+0000 7f10df7fe700 1 -- 172.21.15.123:0/1183807655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d00094f0 con 0x7f10e00634b0 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10df7fe700 1 --2- 172.21.15.123:0/1183807655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10e00634b0 0x7f10e006eb40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f10d400e920 tx=0x7f10d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 1 -- 172.21.15.123:0/1183807655 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4013410 con 0x7f10e00634b0 2023-09-29T23:05:57.122 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 1 -- 172.21.15.123:0/1183807655 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d4013570 con 0x7f10e00634b0 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e009bb10 con 0x7f10e00634b0 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 1 -- 172.21.15.123:0/1183807655 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4013930 con 0x7f10e00634b0 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e006dda0 con 0x7f10e00634b0 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 1 -- 172.21.15.123:0/1183807655 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f10d4013ca0 con 0x7f10e00634b0 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 4 mgrc handle_mgr_map Got map version 1 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 4 mgrc handle_mgr_map Active mgr is now 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 4 mgrc reconnect No active mgr available yet 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.119+0000 7f10c7fff700 1 -- 172.21.15.123:0/1183807655 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f10d401b3b0 con 0x7f10e00634b0 2023-09-29T23:05:57.123 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.120+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e009e5e0 con 0x7f10e00634b0 2023-09-29T23:05:57.124 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.122+0000 7f10c7fff700 1 -- 172.21.15.123:0/1183807655 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f10d403b6d0 con 0x7f10e00634b0 2023-09-29T23:05:57.205 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.205+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8304c4c7-39e6-4a99-bcf5-d39fb111241c", "id": 2} v 0) v1 -- 0x7f10e006ab80 con 0x7f10e00634b0 2023-09-29T23:05:57.207 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.207+0000 7f10c7fff700 1 -- 172.21.15.123:0/1183807655 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8304c4c7-39e6-4a99-bcf5-d39fb111241c", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f10d402a030 con 0x7f10e00634b0 2023-09-29T23:05:57.207 INFO:teuthology.orchestra.run.smithi123.stdout:2 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.208+0000 7f10e6912700 10 mgrc shutdown 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.208+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10e00634b0 msgr2=0x7f10e006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.208+0000 7f10e6912700 1 --2- 172.21.15.123:0/1183807655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10e00634b0 0x7f10e006eb40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f10d400e920 tx=0x7f10d4008040 comp rx=0 tx=0).stop 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.208+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 shutdown_connections 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.208+0000 7f10e6912700 1 --2- 172.21.15.123:0/1183807655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10e00a4570 0x7f10e013ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.208+0000 7f10e6912700 1 --2- 172.21.15.123:0/1183807655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10e00630e0 0x7f10e009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.208+0000 7f10e6912700 1 --2- 172.21.15.123:0/1183807655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10e00634b0 0x7f10e006eb40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.209+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 >> 172.21.15.123:0/1183807655 conn(0x7f10e0069ec0 msgr2=0x7f10e00a5120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.209+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 shutdown_connections 2023-09-29T23:05:57.209 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.209+0000 7f10e6912700 1 -- 172.21.15.123:0/1183807655 wait complete. 2023-09-29T23:05:57.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 0614da14-c1b7-4d44-a6ab-9e49ab25ede0 3 2023-09-29T23:05:57.254 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:57.253+0000 7f4c5ac5f1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-29T23:05:57.378 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:57.379+0000 7f2e3246c1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-29T23:05:57.379 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.379+0000 7f8ccb270700 1 Processor -- start 2023-09-29T23:05:57.379 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8ccb270700 1 -- start start 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8ccb270700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8cc40632a0 0x7f8cc40a4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8ccb270700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc409b4f0 0x7f8cc4140df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8ccb270700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8cc4141330 0x7f8cc4143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8ccb270700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc41417b0 con 0x7f8cc40632a0 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8ccb270700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc409b900 con 0x7f8cc4141330 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8ccb270700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40a4be0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8cc980d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8cc4141330 0x7f8cc4143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.382 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8cc980d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8cc4141330 0x7f8cc4143cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56426/0 (socket says 172.21.15.123:56426) 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8cc980d700 1 -- 172.21.15.123:0/3822664316 learned_addr learned my addr 172.21.15.123:0/3822664316 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8cc900c700 1 --2- 172.21.15.123:0/3822664316 >> v2:172.21.15.121:3300/0 conn(0x7f8cc40632a0 0x7f8cc40a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.380+0000 7f8cc880b700 1 --2- 172.21.15.123:0/3822664316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc409b4f0 0x7f8cc4140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cc980d700 1 -- 172.21.15.123:0/3822664316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc409b4f0 msgr2=0x7f8cc4140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cc980d700 1 --2- 172.21.15.123:0/3822664316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc409b4f0 0x7f8cc4140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cc980d700 1 -- 172.21.15.123:0/3822664316 >> v2:172.21.15.121:3300/0 conn(0x7f8cc40632a0 msgr2=0x7f8cc40a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cc980d700 1 --2- 172.21.15.123:0/3822664316 >> v2:172.21.15.121:3300/0 conn(0x7f8cc40632a0 0x7f8cc40a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cc980d700 1 -- 172.21.15.123:0/3822664316 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40a4ea0 con 0x7f8cc4141330 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cc980d700 1 --2- 172.21.15.123:0/3822664316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 0x7f8cc4143cc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8cc406fe00 tx=0x7f8cbc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c941232b01d68a5b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cbbfff700 1 -- 172.21.15.123:0/3822664316 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc00e070 con 0x7f8cc4141330 2023-09-29T23:05:57.383 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cbbfff700 1 -- 172.21.15.123:0/3822664316 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc00b970 con 0x7f8cc4141330 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.381+0000 7f8cbbfff700 1 -- 172.21.15.123:0/3822664316 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc0092e0 con 0x7f8cc4141330 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 -- 172.21.15.123:0/3822664316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 msgr2=0x7f8cc4143cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 --2- 172.21.15.123:0/3822664316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 0x7f8cc4143cc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8cc406fe00 tx=0x7f8cbc009840 comp rx=0 tx=0).stop 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 -- 172.21.15.123:0/3822664316 shutdown_connections 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 --2- 172.21.15.123:0/3822664316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 0x7f8cc4143cc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 --2- 172.21.15.123:0/3822664316 >> v2:172.21.15.121:3300/0 conn(0x7f8cc40632a0 0x7f8cc40a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 --2- 172.21.15.123:0/3822664316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc409b4f0 0x7f8cc4140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 -- 172.21.15.123:0/3822664316 >> 172.21.15.123:0/3822664316 conn(0x7f8cc4069f10 msgr2=0x7f8cc40b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 -- 172.21.15.123:0/3822664316 shutdown_connections 2023-09-29T23:05:57.384 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.382+0000 7f8ccb270700 1 -- 172.21.15.123:0/3822664316 wait complete. 2023-09-29T23:05:57.386 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 Processor -- start 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 -- start start 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc40632a0 0x7f8cc414ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc409b4f0 0x7f8cc409cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 0x7f8cc409e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4150450 con 0x7f8cc409b4f0 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40a06d0 con 0x7f8cc4141330 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.383+0000 7f8ccb270700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4150680 con 0x7f8cc40632a0 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc409b4f0 0x7f8cc409cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc900c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc40632a0 0x7f8cc414ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc409b4f0 0x7f8cc409cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.123:34886/0 (socket says 172.21.15.123:34886) 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 -- 172.21.15.123:0/4011113659 learned_addr learned my addr 172.21.15.123:0/4011113659 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc980d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 0x7f8cc409e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 -- 172.21.15.123:0/4011113659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc40632a0 msgr2=0x7f8cc414ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 --2- 172.21.15.123:0/4011113659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc40632a0 0x7f8cc414ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 -- 172.21.15.123:0/4011113659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 msgr2=0x7f8cc409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 --2- 172.21.15.123:0/4011113659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 0x7f8cc409e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 -- 172.21.15.123:0/4011113659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc009ae0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cc880b700 1 --2- 172.21.15.123:0/4011113659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc409b4f0 0x7f8cc409cda0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400ec80 tx=0x7f8cb4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 1 -- 172.21.15.123:0/4011113659 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4009680 con 0x7f8cc409b4f0 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 1 -- 172.21.15.123:0/4011113659 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb400c850 con 0x7f8cc409b4f0 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc409c5c0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 1 -- 172.21.15.123:0/4011113659 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400cbd0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc409d2e0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 1 -- 172.21.15.123:0/4011113659 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8cb40136f0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 4 mgrc handle_mgr_map Got map version 1 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 4 mgrc handle_mgr_map Active mgr is now 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 4 mgrc reconnect No active mgr available yet 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.384+0000 7f8cb97fa700 1 -- 172.21.15.123:0/4011113659 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f8cb401c4f0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.385+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc406ab80 con 0x7f8cc409b4f0 2023-09-29T23:05:57.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.387+0000 7f8cb97fa700 1 -- 172.21.15.123:0/4011113659 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8cb403b6d0 con 0x7f8cc409b4f0 2023-09-29T23:05:57.390 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:57.390+0000 7f4c5ac5f1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-29T23:05:57.469 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.469+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0614da14-c1b7-4d44-a6ab-9e49ab25ede0", "id": 3} v 0) v1 -- 0x7f8cc4071660 con 0x7f8cc409b4f0 2023-09-29T23:05:57.471 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.472+0000 7f8cb97fa700 1 -- 172.21.15.123:0/4011113659 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0614da14-c1b7-4d44-a6ab-9e49ab25ede0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8cb402a070 con 0x7f8cc409b4f0 2023-09-29T23:05:57.472 INFO:teuthology.orchestra.run.smithi123.stdout:3 2023-09-29T23:05:57.473 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 10 mgrc shutdown 2023-09-29T23:05:57.473 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc409b4f0 msgr2=0x7f8cc409cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.473 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 --2- 172.21.15.123:0/4011113659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc409b4f0 0x7f8cc409cda0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400ec80 tx=0x7f8cb4009120 comp rx=0 tx=0).stop 2023-09-29T23:05:57.473 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 shutdown_connections 2023-09-29T23:05:57.474 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 --2- 172.21.15.123:0/4011113659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc4141330 0x7f8cc409e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.474 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 --2- 172.21.15.123:0/4011113659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc40632a0 0x7f8cc414ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.474 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 --2- 172.21.15.123:0/4011113659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc409b4f0 0x7f8cc409cda0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.474 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 >> 172.21.15.123:0/4011113659 conn(0x7f8cc4069f10 msgr2=0x7f8cc413eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:57.474 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 shutdown_connections 2023-09-29T23:05:57.474 INFO:teuthology.orchestra.run.smithi123.stderr:2023-09-29T23:05:57.473+0000 7f8ccb270700 1 -- 172.21.15.123:0/4011113659 wait complete. 2023-09-29T23:05:57.484 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-29T23:05:57.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-29T23:05:57.487 INFO:tasks.ceph.osd.0:Started 2023-09-29T23:05:57.487 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-29T23:05:57.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-29T23:05:57.489 INFO:tasks.ceph.osd.1:Started 2023-09-29T23:05:57.489 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-29T23:05:57.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-29T23:05:57.491 INFO:tasks.ceph.osd.2:Started 2023-09-29T23:05:57.491 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-29T23:05:57.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-29T23:05:57.493 INFO:tasks.ceph.osd.3:Started 2023-09-29T23:05:57.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:05:57.527 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:57.527+0000 7fb0d7a7b1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-29T23:05:57.570 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:57.570+0000 7f2e3246c1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-29T23:05:57.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.615+0000 7f82fda48700 1 Processor -- start 2023-09-29T23:05:57.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.615+0000 7f82fda48700 1 -- start start 2023-09-29T23:05:57.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82fda48700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f82f8068200 0x7f82f80a2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82fda48700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f82f80b76e0 0x7f82f80a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82fda48700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f80685d0 con 0x7f82f813fec0 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82fda48700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f80a2670 con 0x7f82f80b76e0 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82fda48700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f80a03e0 con 0x7f82f8068200 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f67fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f82f80b76e0 0x7f82f80a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f77fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f82f813fec0 0x7f82f8143e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52774/0 (socket says 172.21.15.121:52774) 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f77fe700 1 -- 172.21.15.121:0/1116941931 learned_addr learned my addr 172.21.15.121:0/1116941931 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f67fc700 1 --2- 172.21.15.121:0/1116941931 >> v2:172.21.15.121:3300/0 conn(0x7f82f80b76e0 0x7f82f80a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34790/0 (socket says 172.21.15.121:34790) 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f6ffd700 1 --2- 172.21.15.121:0/1116941931 >> v2:172.21.15.123:3300/0 conn(0x7f82f8068200 0x7f82f80a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f67fc700 1 -- 172.21.15.121:0/1116941931 >> v2:172.21.15.123:3300/0 conn(0x7f82f8068200 msgr2=0x7f82f80a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f67fc700 1 --2- 172.21.15.121:0/1116941931 >> v2:172.21.15.123:3300/0 conn(0x7f82f8068200 0x7f82f80a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f67fc700 1 -- 172.21.15.121:0/1116941931 >> v1:172.21.15.121:6789/0 conn(0x7f82f813fec0 legacy=0x7f82f8143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f67fc700 1 -- 172.21.15.121:0/1116941931 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f80a2860 con 0x7f82f80b76e0 2023-09-29T23:05:57.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f5ffb700 1 -- 172.21.15.121:0/1116941931 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541828562 0 0) 0x7f82f80685d0 con 0x7f82f813fec0 2023-09-29T23:05:57.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f67fc700 1 --2- 172.21.15.121:0/1116941931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f80b76e0 0x7f82f80a0d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f82f80688f0 tx=0x7f82ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90e7e96c8fb5c32 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:57.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.616+0000 7f82f5ffb700 1 -- 172.21.15.121:0/1116941931 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec011630 con 0x7f82f80b76e0 2023-09-29T23:05:57.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82f5ffb700 1 -- 172.21.15.121:0/1116941931 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec011790 con 0x7f82f80b76e0 2023-09-29T23:05:57.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82f5ffb700 1 -- 172.21.15.121:0/1116941931 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec011b30 con 0x7f82f80b76e0 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 -- 172.21.15.121:0/1116941931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f80b76e0 msgr2=0x7f82f80a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 --2- 172.21.15.121:0/1116941931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f80b76e0 0x7f82f80a0d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f82f80688f0 tx=0x7f82ec0091a0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 -- 172.21.15.121:0/1116941931 shutdown_connections 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 --2- 172.21.15.121:0/1116941931 >> v2:172.21.15.123:3300/0 conn(0x7f82f8068200 0x7f82f80a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 --2- 172.21.15.121:0/1116941931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f80b76e0 0x7f82f80a0d70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 -- 172.21.15.121:0/1116941931 >> 172.21.15.121:0/1116941931 conn(0x7f82f80a95b0 msgr2=0x7f82f80678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 -- 172.21.15.121:0/1116941931 shutdown_connections 2023-09-29T23:05:57.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.617+0000 7f82fda48700 1 -- 172.21.15.121:0/1116941931 wait complete. 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.619+0000 7f82fda48700 1 Processor -- start 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.619+0000 7f82fda48700 1 -- start start 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.619+0000 7f82fda48700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f8068200 0x7f82f81420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.619+0000 7f82fda48700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82f80b76e0 0x7f82f8139840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.619+0000 7f82fda48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f813fec0 0x7f82f8139d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.619+0000 7f82fda48700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8141b40 con 0x7f82f813fec0 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.619+0000 7f82fda48700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8144c70 con 0x7f82f8068200 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82fda48700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f80b62c0 con 0x7f82f80b76e0 2023-09-29T23:05:57.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82f80b76e0 0x7f82f8139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82f80b76e0 0x7f82f8139840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43720/0 (socket says 172.21.15.121:43720) 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 -- 172.21.15.121:0/2341496671 learned_addr learned my addr 172.21.15.121:0/2341496671 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f77fe700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f813fec0 0x7f82f8139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f6ffd700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f8068200 0x7f82f81420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 -- 172.21.15.121:0/2341496671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f8068200 msgr2=0x7f82f81420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f8068200 0x7f82f81420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 -- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f813fec0 msgr2=0x7f82f8139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f813fec0 0x7f82f8139d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 -- 172.21.15.121:0/2341496671 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009420 con 0x7f82f80b76e0 2023-09-29T23:05:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82f67fc700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82f80b76e0 0x7f82f8139840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f82f8068820 tx=0x7f82ec017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82deffd700 1 -- 172.21.15.121:0/2341496671 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec000d10 con 0x7f82f80b76e0 2023-09-29T23:05:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82deffd700 1 -- 172.21.15.121:0/2341496671 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec011b10 con 0x7f82f80b76e0 2023-09-29T23:05:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82deffd700 1 -- 172.21.15.121:0/2341496671 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec025640 con 0x7f82f80b76e0 2023-09-29T23:05:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f80a6470 con 0x7f82f80b76e0 2023-09-29T23:05:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.620+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f8142a90 con 0x7f82f80b76e0 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.622+0000 7f82deffd700 1 -- 172.21.15.121:0/2341496671 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f82ec025cb0 con 0x7f82f80b76e0 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.622+0000 7f82deffd700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.621+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f8068fb0 con 0x7f82f80b76e0 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.622+0000 7f82deffd700 4 mgrc handle_mgr_map Got map version 1 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.622+0000 7f82deffd700 4 mgrc handle_mgr_map Active mgr is now 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.622+0000 7f82deffd700 4 mgrc reconnect No active mgr available yet 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.622+0000 7f82deffd700 1 -- 172.21.15.121:0/2341496671 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f82ec02edb0 con 0x7f82f80b76e0 2023-09-29T23:05:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.624+0000 7f82deffd700 1 -- 172.21.15.121:0/2341496671 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f82ec033060 con 0x7f82f80b76e0 2023-09-29T23:05:57.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.692+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f8071660 con 0x7f82f80b76e0 2023-09-29T23:05:57.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.693+0000 7f82deffd700 1 -- 172.21.15.121:0/2341496671 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4108 (secure 0 0 0) 0x7f82ec015020 con 0x7f82f80b76e0 2023-09-29T23:05:57.694 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:05:57.694 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":6,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:05:57.470851+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.693+0000 7f82fda48700 10 mgrc shutdown 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.693+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82f80b76e0 msgr2=0x7f82f8139840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.693+0000 7f82fda48700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82f80b76e0 0x7f82f8139840 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f82f8068820 tx=0x7f82ec017000 comp rx=0 tx=0).stop 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.694+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 shutdown_connections 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.694+0000 7f82fda48700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82f813fec0 0x7f82f8139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.694+0000 7f82fda48700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f8068200 0x7f82f81420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.694+0000 7f82fda48700 1 --2- 172.21.15.121:0/2341496671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82f80b76e0 0x7f82f8139840 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:57.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.694+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 >> 172.21.15.121:0/2341496671 conn(0x7f82f80a95b0 msgr2=0x7f82f813cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:57.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.694+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 shutdown_connections 2023-09-29T23:05:57.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:05:57.694+0000 7f82fda48700 1 -- 172.21.15.121:0/2341496671 wait complete. 2023-09-29T23:05:57.702 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:57.703+0000 7fb0d7a7b1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-29T23:05:57.706 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-29T23:05:57.706 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-29T23:05:57.739 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:57.738+0000 7f4c5ac5f1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T23:05:57.935 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:57.935+0000 7f2e3246c1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T23:05:58.046 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:58.046+0000 7fb0d7a7b1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T23:05:58.076 INFO:tasks.ceph.osd.0.smithi121.stderr:2023-09-29T23:05:58.075+0000 7f9f28ac5540 -1 Falling back to public interface 2023-09-29T23:05:58.086 INFO:tasks.ceph.osd.2.smithi123.stderr:2023-09-29T23:05:58.086+0000 7fba1825c540 -1 Falling back to public interface 2023-09-29T23:05:58.092 INFO:tasks.ceph.osd.3.smithi123.stderr:2023-09-29T23:05:58.092+0000 7f6b604aa540 -1 Falling back to public interface 2023-09-29T23:05:58.107 INFO:tasks.ceph.osd.1.smithi123.stderr:2023-09-29T23:05:58.107+0000 7efc91a78540 -1 Falling back to public interface 2023-09-29T23:05:58.109 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-29T23:05:58.218 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 Processor -- start 2023-09-29T23:05:58.219 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 -- start start 2023-09-29T23:05:58.221 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5228070600 0x7f522806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f522813f9f0 0x7f522813fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f52280a20e0 0x7f52280a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52280b40a0 con 0x7f52280a20e0 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522806cd40 con 0x7f522813f9f0 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522e8d2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5228143be0 con 0x7f5228070600 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f5227fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5228070600 0x7f522806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f522ce6f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f52280a20e0 0x7f52280a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:58.222 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f5227fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5228070600 0x7f522806c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43736/0 (socket says 172.21.15.121:43736) 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f5227fff700 1 -- 172.21.15.121:0/902529526 learned_addr learned my addr 172.21.15.121:0/902529526 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.218+0000 7f52277fe700 1 --2- 172.21.15.121:0/902529526 >> v2:172.21.15.123:3300/0 conn(0x7f522813f9f0 0x7f522813fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f522ce6f700 1 -- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5228070600 msgr2=0x7f522806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f522ce6f700 1 --2- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5228070600 0x7f522806c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f522ce6f700 1 -- 172.21.15.121:0/902529526 >> v2:172.21.15.123:3300/0 conn(0x7f522813f9f0 msgr2=0x7f522813fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f522ce6f700 1 --2- 172.21.15.121:0/902529526 >> v2:172.21.15.123:3300/0 conn(0x7f522813f9f0 0x7f522813fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f522ce6f700 1 -- 172.21.15.121:0/902529526 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5228143ea0 con 0x7f52280a20e0 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f5227fff700 1 --2- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5228070600 0x7f522806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:05:58.223 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f52277fe700 1 --2- 172.21.15.121:0/902529526 >> v2:172.21.15.123:3300/0 conn(0x7f522813f9f0 0x7f522813fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f522ce6f700 1 --2- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f52280a2540 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5228070ca0 tx=0x7f5220009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216f07ae7b072989 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f5226ffd700 1 -- 172.21.15.121:0/902529526 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522000e070 con 0x7f52280a20e0 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f5226ffd700 1 -- 172.21.15.121:0/902529526 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522000bbf0 con 0x7f52280a20e0 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.219+0000 7f5226ffd700 1 -- 172.21.15.121:0/902529526 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220012430 con 0x7f52280a20e0 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 -- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 msgr2=0x7f52280a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 --2- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f52280a2540 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5228070ca0 tx=0x7f5220009780 comp rx=0 tx=0).stop 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 -- 172.21.15.121:0/902529526 shutdown_connections 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 --2- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f52280a2540 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 --2- 172.21.15.121:0/902529526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5228070600 0x7f522806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.224 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 --2- 172.21.15.121:0/902529526 >> v2:172.21.15.123:3300/0 conn(0x7f522813f9f0 0x7f522813fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.225 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 -- 172.21.15.121:0/902529526 >> 172.21.15.121:0/902529526 conn(0x7f52280b3230 msgr2=0x7f52280b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:58.225 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 -- 172.21.15.121:0/902529526 shutdown_connections 2023-09-29T23:05:58.225 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.220+0000 7f522e8d2700 1 -- 172.21.15.121:0/902529526 wait complete. 2023-09-29T23:05:58.225 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 Processor -- start 2023-09-29T23:05:58.225 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 -- start start 2023-09-29T23:05:58.227 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5228070600 0x7f522809dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:58.227 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f522806e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:58.227 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522813f9f0 0x7f52281535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:05:58.227 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5228153fa0 con 0x7f52280a20e0 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5228154260 con 0x7f5228070600 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522e8d2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522809e960 con 0x7f522813f9f0 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f52277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f522806e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f522ce6f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522813f9f0 0x7f52281535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f52277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f522806e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34814/0 (socket says 172.21.15.121:34814) 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f52277fe700 1 -- 172.21.15.121:0/1562739748 learned_addr learned my addr 172.21.15.121:0/1562739748 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.221+0000 7f5227fff700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5228070600 0x7f522809dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f52277fe700 1 -- 172.21.15.121:0/1562739748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522813f9f0 msgr2=0x7f52281535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f52277fe700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522813f9f0 0x7f52281535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.228 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f52277fe700 1 -- 172.21.15.121:0/1562739748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5228070600 msgr2=0x7f522809dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f52277fe700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5228070600 0x7f522809dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f52277fe700 1 -- 172.21.15.121:0/1562739748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180094f0 con 0x7f52280a20e0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f52277fe700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f522806e3f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f521c00d860 tx=0x7f521c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 1 -- 172.21.15.121:0/1562739748 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0125f0 con 0x7f52280a20e0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 1 -- 172.21.15.121:0/1562739748 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c012750 con 0x7f52280a20e0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 1 -- 172.21.15.121:0/1562739748 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c012af0 con 0x7f52280a20e0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5220009a20 con 0x7f52280a20e0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522813df40 con 0x7f52280a20e0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 1 -- 172.21.15.121:0/1562739748 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f521c013480 con 0x7f52280a20e0 2023-09-29T23:05:58.229 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-29T23:05:58.230 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 4 mgrc handle_mgr_map Got map version 1 2023-09-29T23:05:58.230 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 4 mgrc handle_mgr_map Active mgr is now 2023-09-29T23:05:58.230 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 4 mgrc reconnect No active mgr available yet 2023-09-29T23:05:58.230 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.222+0000 7f520ffff700 1 -- 172.21.15.121:0/1562739748 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f521c011340 con 0x7f52280a20e0 2023-09-29T23:05:58.230 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.223+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522804c380 con 0x7f52280a20e0 2023-09-29T23:05:58.230 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.227+0000 7f520ffff700 1 -- 172.21.15.121:0/1562739748 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f521c013730 con 0x7f52280a20e0 2023-09-29T23:05:58.295 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.295+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f522809e520 con 0x7f52280a20e0 2023-09-29T23:05:58.296 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.295+0000 7f520ffff700 1 -- 172.21.15.121:0/1562739748 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4108 (secure 0 0 0) 0x7f521c0138d0 con 0x7f52280a20e0 2023-09-29T23:05:58.296 INFO:teuthology.misc.health.smithi121.stdout: 2023-09-29T23:05:58.296 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":6,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:05:57.470851+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:05:58.297 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 10 mgrc shutdown 2023-09-29T23:05:58.297 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 msgr2=0x7f522806e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f522806e3f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f521c00d860 tx=0x7f521c008040 comp rx=0 tx=0).stop 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 shutdown_connections 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522813f9f0 0x7f52281535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5228070600 0x7f522809dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 --2- 172.21.15.121:0/1562739748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52280a20e0 0x7f522806e3f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 >> 172.21.15.121:0/1562739748 conn(0x7f52280b3230 msgr2=0x7f5228142fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 shutdown_connections 2023-09-29T23:05:58.298 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:05:58.296+0000 7f522e8d2700 1 -- 172.21.15.121:0/1562739748 wait complete. 2023-09-29T23:05:58.307 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-09-29T23:05:58.472 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:58.471+0000 7f4c5ac5f1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T23:05:58.543 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:58.542+0000 7f4c5ac5f1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T23:05:58.676 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:58.675+0000 7f4c5ac5f1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T23:05:58.690 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:58.690+0000 7f2e3246c1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T23:05:58.720 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:58.720+0000 7fb0d7a7b1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T23:05:58.742 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:58.742+0000 7f4c5ac5f1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T23:05:58.764 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:58.764+0000 7f2e3246c1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T23:05:58.791 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:58.792+0000 7fb0d7a7b1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T23:05:58.903 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:58.903+0000 7f2e3246c1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T23:05:58.918 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:58.918+0000 7f4c5ac5f1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T23:05:58.921 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:58.921+0000 7fb0d7a7b1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T23:05:58.975 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:58.975+0000 7f2e3246c1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T23:05:58.991 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:58.991+0000 7fb0d7a7b1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T23:05:58.993 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:58.993+0000 7f4c5ac5f1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T23:05:59.068 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:59.067+0000 7f4c5ac5f1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T23:05:59.160 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:59.160+0000 7f4c5ac5f1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T23:05:59.165 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:59.165+0000 7f2e3246c1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T23:05:59.181 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:59.181+0000 7fb0d7a7b1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T23:05:59.246 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:59.246+0000 7f2e3246c1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T23:05:59.258 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:59.258+0000 7fb0d7a7b1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T23:05:59.326 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:59.326+0000 7f2e3246c1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T23:05:59.334 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:59.334+0000 7fb0d7a7b1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T23:05:59.335 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:59.335+0000 7f4c5ac5f1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T23:05:59.414 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:59.413+0000 7f4c5ac5f1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T23:05:59.421 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:59.421+0000 7f2e3246c1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T23:05:59.421 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:59.421+0000 7fb0d7a7b1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T23:05:59.605 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:59.605+0000 7f2e3246c1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T23:05:59.606 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:59.606+0000 7fb0d7a7b1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T23:05:59.688 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:59.688+0000 7f2e3246c1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T23:05:59.689 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:59.689+0000 7fb0d7a7b1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T23:05:59.696 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:59.695+0000 7f4c5ac5f1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T23:05:59.764 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:05:59.764+0000 7f4c5ac5f1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T23:05:59.988 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:05:59.988+0000 7f2e3246c1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T23:05:59.988 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:05:59.988+0000 7fb0d7a7b1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T23:06:00.060 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:00.060+0000 7fb0d7a7b1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T23:06:00.060 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:00.061+0000 7f2e3246c1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T23:06:00.264 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:00.264+0000 7f4c5ac5f1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T23:06:00.575 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:00.575+0000 7f4c5ac5f1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T23:06:00.589 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:00.589+0000 7f2e3246c1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T23:06:00.590 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:00.590+0000 7fb0d7a7b1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T23:06:00.713 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:00.713+0000 7f4c5ac5f1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:06:00.748 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:00.748+0000 7fb0d7a7b1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T23:06:00.971 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:00.971+0000 7f2e3246c1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T23:06:01.078 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:01.077+0000 7f4c5ac5f1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T23:06:01.105 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:01.105+0000 7fb0d7a7b1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:06:01.117 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:01.117+0000 7f2e3246c1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:06:01.219 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:01.219+0000 7f4c5ac5f1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:06:01.367 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:01.366+0000 7f4c5ac5f1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T23:06:01.465 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:01.464+0000 7f4c5ac5f1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T23:06:01.491 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:01.491+0000 7fb0d7a7b1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T23:06:01.507 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:01.492+0000 7f2e3246c1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T23:06:01.507 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:01.506+0000 7f4bea109700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:01.506+0000 7f4bea109700 -1 pg_autoscaler.serve: 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:01.506+0000 7f4bea109700 -1 Traceback (most recent call last): 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr: self._maybe_adjust() 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr: if self.noautoscale: 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:06:01.508 INFO:tasks.ceph.mgr.x.smithi121.stderr: 2023-09-29T23:06:01.563 INFO:tasks.ceph.osd.0.smithi121.stderr:2023-09-29T23:06:01.563+0000 7f9f28ac5540 -1 osd.0 0 log_to_monitors true 2023-09-29T23:06:01.602 INFO:tasks.ceph.osd.2.smithi123.stderr:2023-09-29T23:06:01.602+0000 7fba1825c540 -1 osd.2 0 log_to_monitors true 2023-09-29T23:06:01.626 INFO:tasks.ceph.osd.3.smithi123.stderr:2023-09-29T23:06:01.624+0000 7f6b604aa540 -1 osd.3 0 log_to_monitors true 2023-09-29T23:06:01.627 INFO:tasks.ceph.osd.1.smithi123.stderr:2023-09-29T23:06:01.626+0000 7efc91a78540 -1 osd.1 0 log_to_monitors true 2023-09-29T23:06:01.653 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:01.653+0000 7f2e3246c1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:06:01.654 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:01.655+0000 7fb0d7a7b1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T23:06:01.823 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:01.823+0000 7fb0d7a7b1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T23:06:01.823 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:01.823+0000 7f2e3246c1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T23:06:01.926 INFO:tasks.ceph.mgr.z.smithi123.stderr:2023-09-29T23:06:01.926+0000 7fb0d7a7b1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T23:06:01.927 INFO:tasks.ceph.mgr.y.smithi123.stderr:2023-09-29T23:06:01.927+0000 7f2e3246c1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T23:06:02.493 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:02.493+0000 7f4c1f343700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-29T23:06:02.493 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-09-29T23:06:02.493+0000 7f4c1f343700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-29T23:06:04.711 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:04.823 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.823+0000 7f0b59caa700 1 Processor -- start 2023-09-29T23:06:04.823 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.823+0000 7f0b59caa700 1 -- start start 2023-09-29T23:06:04.828 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b59caa700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0b54070690 0x7f0b540a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:04.828 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b59caa700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0b5413dd30 0x7f0b540a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:04.829 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b59caa700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b54070a60 con 0x7f0b540a2750 2023-09-29T23:06:04.829 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b59caa700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b540a2b90 con 0x7f0b54070690 2023-09-29T23:06:04.829 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b59caa700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b540a1970 con 0x7f0b5413dd30 2023-09-29T23:06:04.829 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0b54070690 0x7f0b540a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:04.829 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0b54070690 0x7f0b540a2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34864/0 (socket says 172.21.15.121:34864) 2023-09-29T23:06:04.829 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 -- 172.21.15.121:0/2515107876 learned_addr learned my addr 172.21.15.121:0/2515107876 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:04.829 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b53fff700 1 --2- 172.21.15.121:0/2515107876 >> v2:172.21.15.123:3300/0 conn(0x7f0b5413dd30 0x7f0b540a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:04.830 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 -- 172.21.15.121:0/2515107876 >> v2:172.21.15.123:3300/0 conn(0x7f0b5413dd30 msgr2=0x7f0b540a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:04.830 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 --2- 172.21.15.121:0/2515107876 >> v2:172.21.15.123:3300/0 conn(0x7f0b5413dd30 0x7f0b540a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:04.830 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 -- 172.21.15.121:0/2515107876 >> v1:172.21.15.123:6789/0 conn(0x7f0b540a2750 legacy=0x7f0b5413d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:04.830 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 -- 172.21.15.121:0/2515107876 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b540a0e60 con 0x7f0b54070690 2023-09-29T23:06:04.830 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b537fe700 1 --2- 172.21.15.121:0/2515107876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b54070690 0x7f0b540a2f90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0b540a5b20 tx=0x7f0b44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0b075a7df5cdee server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:04.830 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.824+0000 7f0b527fc700 1 -- 172.21.15.121:0/2515107876 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b44009e70 con 0x7f0b54070690 2023-09-29T23:06:04.830 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b527fc700 1 -- 172.21.15.121:0/2515107876 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4400b860 con 0x7f0b54070690 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b527fc700 1 -- 172.21.15.121:0/2515107876 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4400bc00 con 0x7f0b54070690 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 -- 172.21.15.121:0/2515107876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b54070690 msgr2=0x7f0b540a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 --2- 172.21.15.121:0/2515107876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b54070690 0x7f0b540a2f90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0b540a5b20 tx=0x7f0b44009250 comp rx=0 tx=0).stop 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 -- 172.21.15.121:0/2515107876 shutdown_connections 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 --2- 172.21.15.121:0/2515107876 >> v2:172.21.15.123:3300/0 conn(0x7f0b5413dd30 0x7f0b540a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 --2- 172.21.15.121:0/2515107876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b54070690 0x7f0b540a2f90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 -- 172.21.15.121:0/2515107876 >> 172.21.15.121:0/2515107876 conn(0x7f0b540b3230 msgr2=0x7f0b540b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 -- 172.21.15.121:0/2515107876 shutdown_connections 2023-09-29T23:06:04.831 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.825+0000 7f0b59caa700 1 -- 172.21.15.121:0/2515107876 wait complete. 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.826+0000 7f0b59caa700 1 Processor -- start 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.826+0000 7f0b59caa700 1 -- start start 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b59caa700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b54070690 0x7f0b54137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b59caa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b540a2750 0x7f0b54142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b59caa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b5413dd30 0x7f0b5413fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b59caa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5413f540 con 0x7f0b5413dd30 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b59caa700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b541431e0 con 0x7f0b540a2750 2023-09-29T23:06:04.832 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b59caa700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5413f730 con 0x7f0b54070690 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b5413dd30 0x7f0b5413fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b537fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b54070690 0x7f0b54137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b5413dd30 0x7f0b5413fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34874/0 (socket says 172.21.15.121:34874) 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b53fff700 1 -- 172.21.15.121:0/1526878961 learned_addr learned my addr 172.21.15.121:0/1526878961 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b52ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b540a2750 0x7f0b54142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b537fe700 1 -- 172.21.15.121:0/1526878961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b540a2750 msgr2=0x7f0b54142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b537fe700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b540a2750 0x7f0b54142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:04.833 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b537fe700 1 -- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b5413dd30 msgr2=0x7f0b5413fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b537fe700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b5413dd30 0x7f0b5413fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b537fe700 1 -- 172.21.15.121:0/1526878961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b440094f0 con 0x7f0b54070690 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b53fff700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b5413dd30 0x7f0b5413fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b537fe700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b54070690 0x7f0b54137d90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b540b4230 tx=0x7f0b44017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/1526878961 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4400bbe0 con 0x7f0b54070690 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.827+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/1526878961 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b44000c90 con 0x7f0b54070690 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.828+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b54138870 con 0x7f0b54070690 2023-09-29T23:06:04.834 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.828+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/1526878961 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b44011650 con 0x7f0b54070690 2023-09-29T23:06:04.835 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.828+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b541382d0 con 0x7f0b54070690 2023-09-29T23:06:04.835 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.828+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b54141330 con 0x7f0b54070690 2023-09-29T23:06:04.836 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/1526878961 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7f0b44011a80 con 0x7f0b54070690 2023-09-29T23:06:04.836 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b3b7fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-29T23:06:04.836 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b3b7fe700 4 mgrc handle_mgr_map Got map version 4 2023-09-29T23:06:04.836 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b3b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:04.836 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b3b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:04.836 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b3b7fe700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0b40098720 0x7f0b4009abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:04.837 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b52ffd700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0b40098720 0x7f0b4009abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:04.837 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.830+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/1526878961 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1619+0+0 (secure 0 0 0) 0x7f0b440c0e50 con 0x7f0b54070690 2023-09-29T23:06:04.837 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.831+0000 7f0b52ffd700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0b40098720 0x7f0b4009abd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0b540a5850 tx=0x7f0b48007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:04.837 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:04.834+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/1526878961 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b4408f100 con 0x7f0b54070690 2023-09-29T23:06:05.030 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.030+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b5406f460 con 0x7f0b54070690 2023-09-29T23:06:05.030 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.031+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/1526878961 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+4108 (secure 0 0 0) 0x7f0b44092720 con 0x7f0b54070690 2023-09-29T23:06:05.031 INFO:teuthology.misc.health.smithi121.stdout: 2023-09-29T23:06:05.031 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":8,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:03.528055+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:05.033 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 10 mgrc shutdown 2023-09-29T23:06:05.033 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0b40098720 msgr2=0x7f0b4009abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:05.033 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0b40098720 0x7f0b4009abd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0b540a5850 tx=0x7f0b48007ec0 comp rx=0 tx=0).stop 2023-09-29T23:06:05.033 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b54070690 msgr2=0x7f0b54137d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:05.033 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b54070690 0x7f0b54137d90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b540b4230 tx=0x7f0b44017000 comp rx=0 tx=0).stop 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 shutdown_connections 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0b40098720 0x7f0b4009abd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b5413dd30 0x7f0b5413fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b54070690 0x7f0b54137d90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 --2- 172.21.15.121:0/1526878961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b540a2750 0x7f0b54142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 >> 172.21.15.121:0/1526878961 conn(0x7f0b540b3230 msgr2=0x7f0b5413a950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 shutdown_connections 2023-09-29T23:06:05.034 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:05.033+0000 7f0b59caa700 1 -- 172.21.15.121:0/1526878961 wait complete. 2023-09-29T23:06:05.044 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-09-29T23:06:06.286 INFO:tasks.ceph.osd.0.smithi121.stderr:2023-09-29T23:06:06.287+0000 7f9f1f6ca700 -1 osd.0 0 waiting for initial osdmap 2023-09-29T23:06:06.289 INFO:tasks.ceph.osd.0.smithi121.stderr:2023-09-29T23:06:06.290+0000 7f9f184c1700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:06:06.473 INFO:tasks.ceph.osd.3.smithi123.stderr:2023-09-29T23:06:06.473+0000 7f6b570af700 -1 osd.3 0 waiting for initial osdmap 2023-09-29T23:06:06.476 INFO:tasks.ceph.osd.3.smithi123.stderr:2023-09-29T23:06:06.476+0000 7f6b4fea6700 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:06:06.516 INFO:tasks.ceph.osd.1.smithi123.stderr:2023-09-29T23:06:06.516+0000 7efc8867d700 -1 osd.1 0 waiting for initial osdmap 2023-09-29T23:06:06.518 INFO:tasks.ceph.osd.1.smithi123.stderr:2023-09-29T23:06:06.518+0000 7efc81474700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:06:06.530 INFO:tasks.ceph.osd.2.smithi123.stderr:2023-09-29T23:06:06.530+0000 7fba0ee61700 -1 osd.2 0 waiting for initial osdmap 2023-09-29T23:06:06.535 INFO:tasks.ceph.osd.2.smithi123.stderr:2023-09-29T23:06:06.536+0000 7fba07c58700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T23:06:07.544 INFO:tasks.ceph.mon.a.smithi121.stderr:2023-09-29T23:06:07.545+0000 7fd17330e700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-29T23:06:11.447 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:11.568 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.568+0000 7f8559e35700 1 Processor -- start 2023-09-29T23:06:11.568 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.568+0000 7f8559e35700 1 -- start start 2023-09-29T23:06:11.570 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8559e35700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8554070690 0x7f85540a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.570 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8559e35700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f855413dd30 0x7f85540a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.570 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8559e35700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540a2b60 con 0x7f85540a2750 2023-09-29T23:06:11.570 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8559e35700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8554070a60 con 0x7f855413dd30 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8559e35700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540a1970 con 0x7f8554070690 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85537fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8554070690 0x7f85540a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8552ffd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f85540a2750 0x7f855413d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44994/0 (socket says 172.21.15.121:44994) 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8552ffd700 1 -- 172.21.15.121:0/3118170206 learned_addr learned my addr 172.21.15.121:0/3118170206 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8553fff700 1 --2- 172.21.15.121:0/3118170206 >> v2:172.21.15.123:3300/0 conn(0x7f855413dd30 0x7f85540a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85527fc700 1 -- 172.21.15.121:0/3118170206 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623899628 0 0) 0x7f8544001110 con 0x7f85540a2750 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85527fc700 1 -- 172.21.15.121:0/3118170206 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f853c003570 con 0x7f85540a2750 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85537fe700 1 -- 172.21.15.121:0/3118170206 >> v2:172.21.15.123:3300/0 conn(0x7f855413dd30 msgr2=0x7f85540a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85537fe700 1 --2- 172.21.15.121:0/3118170206 >> v2:172.21.15.123:3300/0 conn(0x7f855413dd30 0x7f85540a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85537fe700 1 -- 172.21.15.121:0/3118170206 >> v1:172.21.15.121:6789/0 conn(0x7f85540a2750 legacy=0x7f855413d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.571 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85537fe700 1 -- 172.21.15.121:0/3118170206 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85540a0e60 con 0x7f8554070690 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f8553fff700 1 --2- 172.21.15.121:0/3118170206 >> v2:172.21.15.123:3300/0 conn(0x7f855413dd30 0x7f85540a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85537fe700 1 --2- 172.21.15.121:0/3118170206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8554070690 0x7f85540a2f90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f85480098e0 tx=0x7f85480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7efb434ec814ac55 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85527fc700 1 -- 172.21.15.121:0/3118170206 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854800b640 con 0x7f8554070690 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85527fc700 1 -- 172.21.15.121:0/3118170206 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854800b7a0 con 0x7f8554070690 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.569+0000 7f85527fc700 1 -- 172.21.15.121:0/3118170206 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854800bb40 con 0x7f8554070690 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 -- 172.21.15.121:0/3118170206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8554070690 msgr2=0x7f85540a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 --2- 172.21.15.121:0/3118170206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8554070690 0x7f85540a2f90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f85480098e0 tx=0x7f85480091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 -- 172.21.15.121:0/3118170206 shutdown_connections 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 --2- 172.21.15.121:0/3118170206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8554070690 0x7f85540a2f90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 --2- 172.21.15.121:0/3118170206 >> v2:172.21.15.123:3300/0 conn(0x7f855413dd30 0x7f85540a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.572 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 -- 172.21.15.121:0/3118170206 >> 172.21.15.121:0/3118170206 conn(0x7f85540b3230 msgr2=0x7f85540b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:11.573 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 -- 172.21.15.121:0/3118170206 shutdown_connections 2023-09-29T23:06:11.573 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.570+0000 7f8559e35700 1 -- 172.21.15.121:0/3118170206 wait complete. 2023-09-29T23:06:11.575 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.571+0000 7f8559e35700 1 Processor -- start 2023-09-29T23:06:11.575 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.571+0000 7f8559e35700 1 -- start start 2023-09-29T23:06:11.575 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8559e35700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8554070690 0x7f85540a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.575 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8559e35700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540a2750 0x7f8554142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.575 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8559e35700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f855413dd30 0x7f8554137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.575 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8559e35700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554143520 con 0x7f8554070690 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8559e35700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554153dc0 con 0x7f85540a2750 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8559e35700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8554154080 con 0x7f855413dd30 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8554070690 0x7f85540a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8553fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f855413dd30 0x7f8554137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8554070690 0x7f85540a4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35028/0 (socket says 172.21.15.121:35028) 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 -- 172.21.15.121:0/1829494120 learned_addr learned my addr 172.21.15.121:0/1829494120 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8552ffd700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540a2750 0x7f8554142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 -- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f855413dd30 msgr2=0x7f8554137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f855413dd30 0x7f8554137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.576 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 -- 172.21.15.121:0/1829494120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540a2750 msgr2=0x7f8554142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540a2750 0x7f8554142fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 -- 172.21.15.121:0/1829494120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8548009420 con 0x7f8554070690 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f8552ffd700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540a2750 0x7f8554142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.572+0000 7f85537fe700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8554070690 0x7f85540a4900 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f85540a1ab0 tx=0x7f8548017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.573+0000 7f853b7fe700 1 -- 172.21.15.121:0/1829494120 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854801bcf0 con 0x7f8554070690 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.573+0000 7f853b7fe700 1 -- 172.21.15.121:0/1829494120 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854801be90 con 0x7f8554070690 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.573+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8554142ae0 con 0x7f8554070690 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.573+0000 7f853b7fe700 1 -- 172.21.15.121:0/1829494120 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548011670 con 0x7f8554070690 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.573+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8554142380 con 0x7f8554070690 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.574+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8554063b10 con 0x7f8554070690 2023-09-29T23:06:11.577 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f853b7fe700 1 -- 172.21.15.121:0/1829494120 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7f85480119e0 con 0x7f8554070690 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f853b7fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f853b7fe700 4 mgrc handle_mgr_map Got map version 4 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f853b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f853b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f853b7fe700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8540098720 0x7f854009abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f8552ffd700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8540098720 0x7f854009abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f853b7fe700 1 -- 172.21.15.121:0/1829494120 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3125+0+0 (secure 0 0 0) 0x7f85480c11f0 con 0x7f8554070690 2023-09-29T23:06:11.578 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.575+0000 7f8552ffd700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8540098720 0x7f854009abd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f85440058d0 tx=0x7f8544009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:11.579 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.579+0000 7f853b7fe700 1 -- 172.21.15.121:0/1829494120 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f854808eec0 con 0x7f8554070690 2023-09-29T23:06:11.760 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.761+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8554142c70 con 0x7f8554070690 2023-09-29T23:06:11.761 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.761+0000 7f853b7fe700 1 -- 172.21.15.121:0/1829494120 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7519 (secure 0 0 0) 0x7f8554142c70 con 0x7f8554070690 2023-09-29T23:06:11.761 INFO:teuthology.misc.health.smithi121.stdout: 2023-09-29T23:06:11.761 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":12,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:10.294065+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:11.763 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 10 mgrc shutdown 2023-09-29T23:06:11.763 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8540098720 msgr2=0x7f854009abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.763 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8540098720 0x7f854009abd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f85440058d0 tx=0x7f8544009040 comp rx=0 tx=0).stop 2023-09-29T23:06:11.763 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8554070690 msgr2=0x7f85540a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.763 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8554070690 0x7f85540a4900 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f85540a1ab0 tx=0x7f8548017000 comp rx=0 tx=0).stop 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 shutdown_connections 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8540098720 0x7f854009abd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f855413dd30 0x7f8554137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8554070690 0x7f85540a4900 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 --2- 172.21.15.121:0/1829494120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85540a2750 0x7f8554142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 >> 172.21.15.121:0/1829494120 conn(0x7f85540b3230 msgr2=0x7f85540b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 shutdown_connections 2023-09-29T23:06:11.764 INFO:teuthology.misc.health.smithi121.stderr:2023-09-29T23:06:11.763+0000 7f8559e35700 1 -- 172.21.15.121:0/1829494120 wait complete. 2023-09-29T23:06:11.774 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-09-29T23:06:11.774 INFO:tasks.ceph:Creating RBD pool 2023-09-29T23:06:11.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-29T23:06:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.890+0000 7fc7b419c700 1 Processor -- start 2023-09-29T23:06:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.890+0000 7fc7b419c700 1 -- start start 2023-09-29T23:06:11.892 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b419c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc7ac09d2b0 0x7fc7ac0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.892 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b419c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac140890 0x7fc7ac140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.892 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b419c700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac09de60 con 0x7fc7ac0a59c0 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b419c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac09e880 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b419c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac09d680 con 0x7fc7ac140890 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b2739700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac140890 0x7fc7ac140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b1737700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc7ac0a59c0 0x7fc7ac0a5de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44996/0 (socket says 172.21.15.121:44996) 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b1737700 1 -- 172.21.15.121:0/2784066448 learned_addr learned my addr 172.21.15.121:0/2784066448 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b1f38700 1 --2- 172.21.15.121:0/2784066448 >> v2:172.21.15.123:3300/0 conn(0x7fc7ac09d2b0 0x7fc7ac0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b0f36700 1 -- 172.21.15.121:0/2784066448 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632090963 0 0) 0x7fc7ac09de60 con 0x7fc7ac0a59c0 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b2739700 1 -- 172.21.15.121:0/2784066448 >> v2:172.21.15.123:3300/0 conn(0x7fc7ac09d2b0 msgr2=0x7fc7ac0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b2739700 1 --2- 172.21.15.121:0/2784066448 >> v2:172.21.15.123:3300/0 conn(0x7fc7ac09d2b0 0x7fc7ac0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b2739700 1 -- 172.21.15.121:0/2784066448 >> v1:172.21.15.121:6789/0 conn(0x7fc7ac0a59c0 legacy=0x7fc7ac0a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.893 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b2739700 1 -- 172.21.15.121:0/2784066448 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac09e650 con 0x7fc7ac140890 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b2739700 1 --2- 172.21.15.121:0/2784066448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac140890 0x7fc7ac140cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400af70 tx=0x7fc7a400a890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=772af127ebd2f00b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b0f36700 1 -- 172.21.15.121:0/2784066448 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a400f070 con 0x7fc7ac140890 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b0f36700 1 -- 172.21.15.121:0/2784066448 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a400a2e0 con 0x7fc7ac140890 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.891+0000 7fc7b0f36700 1 -- 172.21.15.121:0/2784066448 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a400cd20 con 0x7fc7ac140890 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 -- 172.21.15.121:0/2784066448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac140890 msgr2=0x7fc7ac140cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2784066448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac140890 0x7fc7ac140cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400af70 tx=0x7fc7a400a890 comp rx=0 tx=0).stop 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 -- 172.21.15.121:0/2784066448 shutdown_connections 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2784066448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac140890 0x7fc7ac140cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2784066448 >> v2:172.21.15.123:3300/0 conn(0x7fc7ac09d2b0 0x7fc7ac0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 -- 172.21.15.121:0/2784066448 >> 172.21.15.121:0/2784066448 conn(0x7fc7ac069ec0 msgr2=0x7fc7ac0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:11.894 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 -- 172.21.15.121:0/2784066448 shutdown_connections 2023-09-29T23:06:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.892+0000 7fc7b419c700 1 -- 172.21.15.121:0/2784066448 wait complete. 2023-09-29T23:06:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 Processor -- start 2023-09-29T23:06:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 -- start start 2023-09-29T23:06:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac09d2b0 0x7fc7ac143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0a59c0 0x7fc7ac151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7ac140890 0x7fc7ac151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac151eb0 con 0x7fc7ac0a59c0 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac099d20 con 0x7fc7ac140890 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.893+0000 7fc7b419c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac1520c0 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac09d2b0 0x7fc7ac143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac09d2b0 0x7fc7ac143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50068/0 (socket says 172.21.15.121:50068) 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1737700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0a59c0 0x7fc7ac151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 -- 172.21.15.121:0/2976751330 learned_addr learned my addr 172.21.15.121:0/2976751330 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b2739700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7ac140890 0x7fc7ac151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 -- 172.21.15.121:0/2976751330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7ac140890 msgr2=0x7fc7ac151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7ac140890 0x7fc7ac151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 -- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0a59c0 msgr2=0x7fc7ac151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0a59c0 0x7fc7ac151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 -- 172.21.15.121:0/2976751330 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a400ab30 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b1f38700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac09d2b0 0x7fc7ac143570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00d890 tx=0x7fc79c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c0125f0 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c012750 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac13ef50 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c012b10 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.894+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac095f30 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.895+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac06ab80 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.896+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7fc79c01e460 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.896+0000 7fc7a1ffb700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-29T23:06:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.897+0000 7fc7a1ffb700 4 mgrc handle_mgr_map Got map version 4 2023-09-29T23:06:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.897+0000 7fc7a1ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.897+0000 7fc7a1ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.897+0000 7fc7a1ffb700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc7900986f0 0x7fc79009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.897+0000 7fc7b1737700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc7900986f0 0x7fc79009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.897+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3125+0+0 (secure 0 0 0) 0x7fc79c0c0330 con 0x7fc7ac09d2b0 2023-09-29T23:06:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.897+0000 7fc7b1737700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc7900986f0 0x7fc79009aba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac143350 tx=0x7fc7a8007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:11.900 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:11.901+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc79c08df60 con 0x7fc7ac09d2b0 2023-09-29T23:06:12.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.084+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fc7ac071660 con 0x7fc7ac09d2b0 2023-09-29T23:06:12.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.377+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc79c08e330 con 0x7fc7ac09d2b0 2023-09-29T23:06:12.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.377+0000 7fc7a1ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:12.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.378+0000 7fc7a1ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:12.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.378+0000 7fc7a1ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:12.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.378+0000 7fc7a1ffb700 1 -- 172.21.15.121:0/2976751330 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7fc79c0b87a0 con 0x7fc7ac09d2b0 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 10 mgrc shutdown 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc7900986f0 msgr2=0x7fc79009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc7900986f0 0x7fc79009aba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac143350 tx=0x7fc7a8007d30 comp rx=0 tx=0).stop 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac09d2b0 msgr2=0x7fc7ac143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac09d2b0 0x7fc7ac143570 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00d890 tx=0x7fc79c008040 comp rx=0 tx=0).stop 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 shutdown_connections 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc7900986f0 0x7fc79009aba0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7ac140890 0x7fc7ac151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7ac09d2b0 0x7fc7ac143570 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 --2- 172.21.15.121:0/2976751330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7ac0a59c0 0x7fc7ac151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 >> 172.21.15.121:0/2976751330 conn(0x7fc7ac069ec0 msgr2=0x7fc7ac0a42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:12.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 shutdown_connections 2023-09-29T23:06:12.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.381+0000 7fc7b419c700 1 -- 172.21.15.121:0/2976751330 wait complete. 2023-09-29T23:06:12.382 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'rbd' created 2023-09-29T23:06:12.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-29T23:06:12.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.511+0000 7fe765df8700 1 Processor -- start 2023-09-29T23:06:12.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.511+0000 7fe765df8700 1 -- start start 2023-09-29T23:06:12.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe765df8700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe76013ded0 0x7fe76009e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe765df8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe765df8700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7601418b0 0x7fe760141d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe765df8700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe760142270 con 0x7fe7601418b0 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe765df8700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7600b26c0 con 0x7fe76013ded0 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe765df8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76009e900 con 0x7fe7600b5760 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7601418b0 0x7fe760141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7601418b0 0x7fe760141d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35054/0 (socket says 172.21.15.121:35054) 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 -- 172.21.15.121:0/4229467898 learned_addr learned my addr 172.21.15.121:0/4229467898 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:12.515 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50082/0 (socket says 172.21.15.121:50082) 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75f7fe700 1 --2- 172.21.15.121:0/4229467898 >> v2:172.21.15.123:3300/0 conn(0x7fe76013ded0 0x7fe76009e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 -- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 msgr2=0x7fe7600b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 --2- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 -- 172.21.15.121:0/4229467898 >> v2:172.21.15.123:3300/0 conn(0x7fe76013ded0 msgr2=0x7fe76009e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 --2- 172.21.15.121:0/4229467898 >> v2:172.21.15.123:3300/0 conn(0x7fe76013ded0 0x7fe76009e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75ffff700 1 -- 172.21.15.121:0/4229467898 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760142760 con 0x7fe7601418b0 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75effd700 1 --2- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.512+0000 7fe75f7fe700 1 --2- 172.21.15.121:0/4229467898 >> v2:172.21.15.123:3300/0 conn(0x7fe76013ded0 0x7fe76009e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe75ffff700 1 --2- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601418b0 0x7fe760141d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe76006f1c0 tx=0x7fe758009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5e54106c4c2271 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe75e7fc700 1 -- 172.21.15.121:0/4229467898 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75800e070 con 0x7fe7601418b0 2023-09-29T23:06:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe75e7fc700 1 -- 172.21.15.121:0/4229467898 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75800bab0 con 0x7fe7601418b0 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe75e7fc700 1 -- 172.21.15.121:0/4229467898 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580092e0 con 0x7fe7601418b0 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 -- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601418b0 msgr2=0x7fe760141d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 --2- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601418b0 0x7fe760141d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe76006f1c0 tx=0x7fe758009840 comp rx=0 tx=0).stop 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 -- 172.21.15.121:0/4229467898 shutdown_connections 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 --2- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601418b0 0x7fe760141d30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 --2- 172.21.15.121:0/4229467898 >> v2:172.21.15.123:3300/0 conn(0x7fe76013ded0 0x7fe76009e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 --2- 172.21.15.121:0/4229467898 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 -- 172.21.15.121:0/4229467898 >> 172.21.15.121:0/4229467898 conn(0x7fe76006d5b0 msgr2=0x7fe760067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 -- 172.21.15.121:0/4229467898 shutdown_connections 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.513+0000 7fe765df8700 1 -- 172.21.15.121:0/4229467898 wait complete. 2023-09-29T23:06:12.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 Processor -- start 2023-09-29T23:06:12.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 -- start start 2023-09-29T23:06:12.520 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:12.520 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7601418b0 0x7fe7600a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:12.520 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601535a0 0x7fe760153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:12.520 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600a2430 con 0x7fe7601535a0 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600a1040 con 0x7fe7601418b0 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe765df8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76014fcf0 con 0x7fe7600b5760 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe75f7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe75ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601535a0 0x7fe760153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe75f7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600a6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50090/0 (socket says 172.21.15.121:50090) 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe75ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601535a0 0x7fe760153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35070/0 (socket says 172.21.15.121:35070) 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe75f7fe700 1 -- 172.21.15.121:0/1309632948 learned_addr learned my addr 172.21.15.121:0/1309632948 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.515+0000 7fe75f7fe700 1 -- 172.21.15.121:0/1309632948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7601418b0 msgr2=0x7fe7600a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe75f7fe700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7601418b0 0x7fe7600a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe75f7fe700 1 -- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601535a0 msgr2=0x7fe760153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:12.521 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe75f7fe700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601535a0 0x7fe760153a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe75f7fe700 1 -- 172.21.15.121:0/1309632948 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe750009420 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe75ffff700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601535a0 0x7fe760153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe75f7fe700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600a6850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c860 tx=0x7fe748008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe7577fe700 1 -- 172.21.15.121:0/1309632948 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748011600 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe7577fe700 1 -- 172.21.15.121:0/1309632948 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe748011760 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe7577fe700 1 -- 172.21.15.121:0/1309632948 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748011b20 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe758009ae0 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.516+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe760069740 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.517+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe760153270 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.518+0000 7fe7577fe700 1 -- 172.21.15.121:0/1309632948 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe7480193b0 con 0x7fe7600b5760 2023-09-29T23:06:12.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.518+0000 7fe7577fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.518+0000 7fe7577fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.518+0000 7fe7577fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.518+0000 7fe7577fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.518+0000 7fe7577fe700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe740098780 0x7fe74009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.519+0000 7fe75effd700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe740098780 0x7fe74009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.519+0000 7fe7577fe700 1 -- 172.21.15.121:0/1309632948 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3467+0+0 (secure 0 0 0) 0x7fe7480c0120 con 0x7fe7600b5760 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.519+0000 7fe75effd700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe740098780 0x7fe74009ac30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe760065790 tx=0x7fe7500091a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:12.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.523+0000 7fe7577fe700 1 -- 172.21.15.121:0/1309632948 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe74808dbc0 con 0x7fe7600b5760 2023-09-29T23:06:12.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:12.706+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe760064c00 con 0x7fe7600b5760 2023-09-29T23:06:13.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.376+0000 7fe7577fe700 1 -- 172.21.15.121:0/1309632948 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe7480150c0 con 0x7fe7600b5760 2023-09-29T23:06:13.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.378+0000 7fe765df8700 10 mgrc shutdown 2023-09-29T23:06:13.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe740098780 msgr2=0x7fe74009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe740098780 0x7fe74009ac30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe760065790 tx=0x7fe7500091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 msgr2=0x7fe7600a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600a6850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c860 tx=0x7fe748008040 comp rx=0 tx=0).stop 2023-09-29T23:06:13.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 shutdown_connections 2023-09-29T23:06:13.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe740098780 0x7fe74009ac30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7601535a0 0x7fe760153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7600b5760 0x7fe7600a6850 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 --2- 172.21.15.121:0/1309632948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7601418b0 0x7fe7600a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.379+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 >> 172.21.15.121:0/1309632948 conn(0x7fe76006d5b0 msgr2=0x7fe76006ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:13.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.380+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 shutdown_connections 2023-09-29T23:06:13.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.380+0000 7fe765df8700 1 -- 172.21.15.121:0/1309632948 wait complete. 2023-09-29T23:06:13.381 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-29T23:06:13.394 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-29T23:06:13.394 INFO:tasks.ceph.mds.a:Restarting daemon 2023-09-29T23:06:13.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-09-29T23:06:13.396 INFO:tasks.ceph.mds.a:Started 2023-09-29T23:06:13.397 INFO:tasks.ceph.mds.c:Restarting daemon 2023-09-29T23:06:13.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-09-29T23:06:13.399 INFO:tasks.ceph.mds.c:Started 2023-09-29T23:06:13.399 INFO:tasks.ceph.mds.b:Restarting daemon 2023-09-29T23:06:13.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-09-29T23:06:13.401 INFO:tasks.ceph.mds.b:Started 2023-09-29T23:06:13.402 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-09-29T23:06:13.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:13.476 INFO:tasks.ceph.mds.c.smithi121.stdout:starting mds.c at 2023-09-29T23:06:13.476 INFO:tasks.ceph.mds.a.smithi121.stdout:starting mds.a at 2023-09-29T23:06:13.484 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-09-29T23:06:13.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.530+0000 7f837c1d7700 1 Processor -- start 2023-09-29T23:06:13.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.530+0000 7f837c1d7700 1 -- start start 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f837c1d7700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8374063330 0x7f83740a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f837c1d7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f83740a2a70 0x7f83740a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f837c1d7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8374063700 con 0x7f837409f820 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f837c1d7700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83740a2ed0 con 0x7f8374063330 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f837c1d7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83740a57e0 con 0x7f83740a2a70 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f8379f73700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8374063330 0x7f83740a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f8379f73700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8374063330 0x7f83740a35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35086/0 (socket says 172.21.15.121:35086) 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f8379f73700 1 -- 172.21.15.121:0/4005399433 learned_addr learned my addr 172.21.15.121:0/4005399433 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f837a774700 1 --2- 172.21.15.121:0/4005399433 >> v2:172.21.15.123:3300/0 conn(0x7f83740a2a70 0x7f83740a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f8379f73700 1 -- 172.21.15.121:0/4005399433 >> v2:172.21.15.123:3300/0 conn(0x7f83740a2a70 msgr2=0x7f83740a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:13.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.531+0000 7f8379f73700 1 --2- 172.21.15.121:0/4005399433 >> v2:172.21.15.123:3300/0 conn(0x7f83740a2a70 0x7f83740a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f8379f73700 1 -- 172.21.15.121:0/4005399433 >> v1:172.21.15.123:6789/0 conn(0x7f837409f820 legacy=0x7f83740a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f8379f73700 1 -- 172.21.15.121:0/4005399433 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83740a0880 con 0x7f8374063330 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f8379f73700 1 --2- 172.21.15.121:0/4005399433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8374063330 0x7f83740a35c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f83740a3110 tx=0x7f8370009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d666242c3e0be1e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f8378f71700 1 -- 172.21.15.121:0/4005399433 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370005c10 con 0x7f8374063330 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f8378f71700 1 -- 172.21.15.121:0/4005399433 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370005d70 con 0x7f8374063330 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f8378f71700 1 -- 172.21.15.121:0/4005399433 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837000b840 con 0x7f8374063330 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 -- 172.21.15.121:0/4005399433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8374063330 msgr2=0x7f83740a35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 --2- 172.21.15.121:0/4005399433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8374063330 0x7f83740a35c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f83740a3110 tx=0x7f8370009220 comp rx=0 tx=0).stop 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 -- 172.21.15.121:0/4005399433 shutdown_connections 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 --2- 172.21.15.121:0/4005399433 >> v2:172.21.15.123:3300/0 conn(0x7f83740a2a70 0x7f83740a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 --2- 172.21.15.121:0/4005399433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8374063330 0x7f83740a35c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 -- 172.21.15.121:0/4005399433 >> 172.21.15.121:0/4005399433 conn(0x7f83740a5bc0 msgr2=0x7f83740b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:13.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 -- 172.21.15.121:0/4005399433 shutdown_connections 2023-09-29T23:06:13.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.532+0000 7f837c1d7700 1 -- 172.21.15.121:0/4005399433 wait complete. 2023-09-29T23:06:13.537 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.533+0000 7f837c1d7700 1 Processor -- start 2023-09-29T23:06:13.537 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.533+0000 7f837c1d7700 1 -- start start 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837c1d7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8374063330 0x7f83740a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837c1d7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f837409f820 0x7f8374150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837c1d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83740a2a70 0x7f8374151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837c1d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83741516b0 con 0x7f83740a2a70 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837c1d7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83740a9610 con 0x7f837409f820 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837c1d7700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837409cad0 con 0x7f8374063330 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f8379f73700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8374063330 0x7f83740a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837a774700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83740a2a70 0x7f8374151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f8379f73700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8374063330 0x7f83740a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50134/0 (socket says 172.21.15.121:50134) 2023-09-29T23:06:13.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f8379f73700 1 -- 172.21.15.121:0/2262110471 learned_addr learned my addr 172.21.15.121:0/2262110471 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f8379772700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f837409f820 0x7f8374150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837a774700 1 -- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8374063330 msgr2=0x7f83740a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837a774700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8374063330 0x7f83740a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837a774700 1 -- 172.21.15.121:0/2262110471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f837409f820 msgr2=0x7f8374150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837a774700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f837409f820 0x7f8374150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837a774700 1 -- 172.21.15.121:0/2262110471 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8370009480 con 0x7f83740a2a70 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f8379f73700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8374063330 0x7f83740a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.534+0000 7f837a774700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83740a2a70 0x7f8374151140 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f836c007e70 tx=0x7f836c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.535+0000 7f8369ffb700 1 -- 172.21.15.121:0/2262110471 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c018070 con 0x7f83740a2a70 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.535+0000 7f8369ffb700 1 -- 172.21.15.121:0/2262110471 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836c00eac0 con 0x7f83740a2a70 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.535+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83741506a0 con 0x7f83740a2a70 2023-09-29T23:06:13.539 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.535+0000 7f8369ffb700 1 -- 172.21.15.121:0/2262110471 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c00ee60 con 0x7f83740a2a70 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.535+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837409ea50 con 0x7f83740a2a70 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.536+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837406ecf0 con 0x7f83740a2a70 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8369ffb700 1 -- 172.21.15.121:0/2262110471 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f836c003020 con 0x7f83740a2a70 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8369ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8369ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8369ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8369ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8369ffb700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8358098780 0x7f835809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8369ffb700 1 -- 172.21.15.121:0/2262110471 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f836c0c10a0 con 0x7f83740a2a70 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.537+0000 7f8379f73700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8358098780 0x7f835809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.538+0000 7f8379f73700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8358098780 0x7f835809ac30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8370007fd0 tx=0x7f8370015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:13.541 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.541+0000 7f8369ffb700 1 -- 172.21.15.121:0/2262110471 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f836c092530 con 0x7f83740a2a70 2023-09-29T23:06:13.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.723+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f837406abd0 con 0x7f83740a2a70 2023-09-29T23:06:13.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.723+0000 7f8369ffb700 1 -- 172.21.15.121:0/2262110471 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9050 (secure 0 0 0) 0x7f836c0098f0 con 0x7f83740a2a70 2023-09-29T23:06:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:13.723 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":14,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:13.368107+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 10 mgrc shutdown 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8358098780 msgr2=0x7f835809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8358098780 0x7f835809ac30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8370007fd0 tx=0x7f8370015000 comp rx=0 tx=0).stop 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83740a2a70 msgr2=0x7f8374151140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83740a2a70 0x7f8374151140 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f836c007e70 tx=0x7f836c0097b0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 shutdown_connections 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8358098780 0x7f835809ac30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83740a2a70 0x7f8374151140 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8374063330 0x7f83740a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 --2- 172.21.15.121:0/2262110471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f837409f820 0x7f8374150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 >> 172.21.15.121:0/2262110471 conn(0x7f83740a5bc0 msgr2=0x7f837409eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:13.727 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.725+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 shutdown_connections 2023-09-29T23:06:13.727 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.726+0000 7f837c1d7700 1 -- 172.21.15.121:0/2262110471 wait complete. 2023-09-29T23:06:13.737 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-29T23:06:07.527057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-09-29T23:06:12.101557+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-09-29T23:06:13.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-29T23:06:13.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.857+0000 7f44de422700 1 Processor -- start 2023-09-29T23:06:13.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.857+0000 7f44de422700 1 -- start start 2023-09-29T23:06:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.857+0000 7f44de422700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44d806af00 0x7f44d8141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.857+0000 7f44de422700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d809e8c0 0x7f44d8142510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.857+0000 7f44de422700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80b5090 con 0x7f44d81344a0 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.857+0000 7f44de422700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d806b2d0 con 0x7f44d806af00 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.857+0000 7f44de422700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80b3460 con 0x7f44d809e8c0 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44d806af00 0x7f44d8141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d77fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d809e8c0 0x7f44d8142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44d806af00 0x7f44d8141fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35094/0 (socket says 172.21.15.121:35094) 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 -- 172.21.15.121:0/3327995695 learned_addr learned my addr 172.21.15.121:0/3327995695 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 -- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d809e8c0 msgr2=0x7f44d8142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 --2- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d809e8c0 0x7f44d8142510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 -- 172.21.15.121:0/3327995695 >> v1:172.21.15.123:6789/0 conn(0x7f44d81344a0 legacy=0x7f44d806e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 -- 172.21.15.121:0/3327995695 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d80b3720 con 0x7f44d806af00 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d77fe700 1 --2- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d809e8c0 0x7f44d8142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d7fff700 1 --2- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d806af00 0x7f44d8141fd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009990 tx=0x7f44c8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c61ac6ecf8fe70 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d6ffd700 1 -- 172.21.15.121:0/3327995695 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8009e80 con 0x7f44d806af00 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d6ffd700 1 -- 172.21.15.121:0/3327995695 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c8005c10 con 0x7f44d806af00 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.858+0000 7f44d6ffd700 1 -- 172.21.15.121:0/3327995695 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c800b650 con 0x7f44d806af00 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 -- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d806af00 msgr2=0x7f44d8141fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 --2- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d806af00 0x7f44d8141fd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009990 tx=0x7f44c8009220 comp rx=0 tx=0).stop 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 -- 172.21.15.121:0/3327995695 shutdown_connections 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 --2- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d806af00 0x7f44d8141fd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 --2- 172.21.15.121:0/3327995695 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d809e8c0 0x7f44d8142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 -- 172.21.15.121:0/3327995695 >> 172.21.15.121:0/3327995695 conn(0x7f44d806d160 msgr2=0x7f44d806d530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:13.861 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 -- 172.21.15.121:0/3327995695 shutdown_connections 2023-09-29T23:06:13.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.859+0000 7f44de422700 1 -- 172.21.15.121:0/3327995695 wait complete. 2023-09-29T23:06:13.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 Processor -- start 2023-09-29T23:06:13.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 -- start start 2023-09-29T23:06:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d806af00 0x7f44d80a3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d809e8c0 0x7f44d809a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44d81344a0 0x7f44d8068550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8069fb0 con 0x7f44d809e8c0 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d809b480 con 0x7f44d81344a0 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44de422700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d809b760 con 0x7f44d806af00 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d806af00 0x7f44d80a3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d806af00 0x7f44d80a3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50148/0 (socket says 172.21.15.121:50148) 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44d7fff700 1 -- 172.21.15.121:0/776824825 learned_addr learned my addr 172.21.15.121:0/776824825 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.860+0000 7f44d77fe700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d809e8c0 0x7f44d809a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44dc9bf700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44d81344a0 0x7f44d8068550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44d7fff700 1 -- 172.21.15.121:0/776824825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44d81344a0 msgr2=0x7f44d8068550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44d7fff700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44d81344a0 0x7f44d8068550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44d7fff700 1 -- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d809e8c0 msgr2=0x7f44d809a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44d7fff700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d809e8c0 0x7f44d809a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44d7fff700 1 -- 172.21.15.121:0/776824825 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc009420 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44d77fe700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d809e8c0 0x7f44d809a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44d7fff700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d806af00 0x7f44d80a3b30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f44c800b3c0 tx=0x7f44c8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44bffff700 1 -- 172.21.15.121:0/776824825 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8011c20 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44bffff700 1 -- 172.21.15.121:0/776824825 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c8011d80 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c8009480 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44bffff700 1 -- 172.21.15.121:0/776824825 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c801c660 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.861+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d809af20 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.862+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d809b920 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.863+0000 7f44bffff700 1 -- 172.21.15.121:0/776824825 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f44c801ca10 con 0x7f44d806af00 2023-09-29T23:06:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.863+0000 7f44bffff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.863+0000 7f44bffff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.863+0000 7f44bffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.863+0000 7f44bffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.863+0000 7f44bffff700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f44c4098830 0x7f44c409ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.864+0000 7f44d77fe700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f44c4098830 0x7f44c409ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.864+0000 7f44bffff700 1 -- 172.21.15.121:0/776824825 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f44c80c1990 con 0x7f44d806af00 2023-09-29T23:06:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.864+0000 7f44d77fe700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f44c4098830 0x7f44c409ace0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44d8134410 tx=0x7f44cc0091a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:13.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:13.867+0000 7f44bffff700 1 -- 172.21.15.121:0/776824825 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f44c808f380 con 0x7f44d806af00 2023-09-29T23:06:14.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.048+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44d8071660 con 0x7f44d806af00 2023-09-29T23:06:14.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.048+0000 7f44bffff700 1 -- 172.21.15.121:0/776824825 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v14) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44c80929a0 con 0x7f44d806af00 2023-09-29T23:06:14.048 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2023-09-29T23:06:14.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 10 mgrc shutdown 2023-09-29T23:06:14.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f44c4098830 msgr2=0x7f44c409ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f44c4098830 0x7f44c409ace0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44d8134410 tx=0x7f44cc0091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d806af00 msgr2=0x7f44d80a3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d806af00 0x7f44d80a3b30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f44c800b3c0 tx=0x7f44c8016000 comp rx=0 tx=0).stop 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 shutdown_connections 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f44c4098830 0x7f44c409ace0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44d81344a0 0x7f44d8068550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d806af00 0x7f44d80a3b30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 --2- 172.21.15.121:0/776824825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d809e8c0 0x7f44d809a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 >> 172.21.15.121:0/776824825 conn(0x7f44d806d160 msgr2=0x7f44d80a4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 shutdown_connections 2023-09-29T23:06:14.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.050+0000 7f44de422700 1 -- 172.21.15.121:0/776824825 wait complete. 2023-09-29T23:06:14.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-09-29T23:06:14.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.181+0000 7fc3f4a88700 1 Processor -- start 2023-09-29T23:06:14.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.181+0000 7fc3f4a88700 1 -- start start 2023-09-29T23:06:14.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3f4a88700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc3f006acb0 0x7fc3f00b4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3f4a88700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3f00b3a90 0x7fc3f0143a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3f4a88700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f00b4870 con 0x7fc3f006b080 2023-09-29T23:06:14.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3f4a88700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f0143fc0 con 0x7fc3f006acb0 2023-09-29T23:06:14.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3f4a88700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f00b4b30 con 0x7fc3f00b3a90 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3ee59c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc3f006acb0 0x7fc3f00b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3ee59c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc3f006acb0 0x7fc3f00b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35108/0 (socket says 172.21.15.121:35108) 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3ee59c700 1 -- 172.21.15.121:0/2771158657 learned_addr learned my addr 172.21.15.121:0/2771158657 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3eed9d700 1 --2- 172.21.15.121:0/2771158657 >> v2:172.21.15.123:3300/0 conn(0x7fc3f00b3a90 0x7fc3f0143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3ee59c700 1 -- 172.21.15.121:0/2771158657 >> v2:172.21.15.123:3300/0 conn(0x7fc3f00b3a90 msgr2=0x7fc3f0143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3ee59c700 1 --2- 172.21.15.121:0/2771158657 >> v2:172.21.15.123:3300/0 conn(0x7fc3f00b3a90 0x7fc3f0143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.182+0000 7fc3ee59c700 1 -- 172.21.15.121:0/2771158657 >> v1:172.21.15.123:6789/0 conn(0x7fc3f006b080 legacy=0x7fc3f0144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3ee59c700 1 -- 172.21.15.121:0/2771158657 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f0069be0 con 0x7fc3f006acb0 2023-09-29T23:06:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3eed9d700 1 --2- 172.21.15.121:0/2771158657 >> v2:172.21.15.123:3300/0 conn(0x7fc3f00b3a90 0x7fc3f0143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3ee59c700 1 --2- 172.21.15.121:0/2771158657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f006acb0 0x7fc3f00b4270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00b5450 tx=0x7fc3e4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e1895eb27e0c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3ed59a700 1 -- 172.21.15.121:0/2771158657 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4005c10 con 0x7fc3f006acb0 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3ed59a700 1 -- 172.21.15.121:0/2771158657 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4005d70 con 0x7fc3f006acb0 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3ed59a700 1 -- 172.21.15.121:0/2771158657 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e400b840 con 0x7fc3f006acb0 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 -- 172.21.15.121:0/2771158657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f006acb0 msgr2=0x7fc3f00b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/2771158657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f006acb0 0x7fc3f00b4270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00b5450 tx=0x7fc3e4009220 comp rx=0 tx=0).stop 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 -- 172.21.15.121:0/2771158657 shutdown_connections 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/2771158657 >> v2:172.21.15.123:3300/0 conn(0x7fc3f00b3a90 0x7fc3f0143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/2771158657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f006acb0 0x7fc3f00b4270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 -- 172.21.15.121:0/2771158657 >> 172.21.15.121:0/2771158657 conn(0x7fc3f006d160 msgr2=0x7fc3f006d560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 -- 172.21.15.121:0/2771158657 shutdown_connections 2023-09-29T23:06:14.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.183+0000 7fc3f4a88700 1 -- 172.21.15.121:0/2771158657 wait complete. 2023-09-29T23:06:14.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.184+0000 7fc3f4a88700 1 Processor -- start 2023-09-29T23:06:14.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.184+0000 7fc3f4a88700 1 -- start start 2023-09-29T23:06:14.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3f4a88700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3f006acb0 0x7fc3f00a9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3f4a88700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006b080 0x7fc3f0142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3f4a88700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f00b3a90 0x7fc3f0150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3f4a88700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f0150200 con 0x7fc3f00b3a90 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3f4a88700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f00a9640 con 0x7fc3f006b080 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3f4a88700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f0151050 con 0x7fc3f006acb0 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f00b3a90 0x7fc3f0150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3ee59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3f006acb0 0x7fc3f00a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f00b3a90 0x7fc3f0150680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35110/0 (socket says 172.21.15.121:35110) 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 -- 172.21.15.121:0/1260150529 learned_addr learned my addr 172.21.15.121:0/1260150529 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3edd9b700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006b080 0x7fc3f0142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 -- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3f006acb0 msgr2=0x7fc3f00a9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3f006acb0 0x7fc3f00a9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 -- 172.21.15.121:0/1260150529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006b080 msgr2=0x7fc3f0142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006b080 0x7fc3f0142b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3eed9d700 1 -- 172.21.15.121:0/1260150529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e4009480 con 0x7fc3f00b3a90 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.185+0000 7fc3edd9b700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006b080 0x7fc3f0142b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.186+0000 7fc3eed9d700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f00b3a90 0x7fc3f0150680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800cef0 tx=0x7fc3e8007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.186+0000 7fc3de7fc700 1 -- 172.21.15.121:0/1260150529 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e800add0 con 0x7fc3f00b3a90 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.186+0000 7fc3de7fc700 1 -- 172.21.15.121:0/1260150529 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e8016420 con 0x7fc3f00b3a90 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.186+0000 7fc3de7fc700 1 -- 172.21.15.121:0/1260150529 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e80167e0 con 0x7fc3f00b3a90 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.186+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f0151310 con 0x7fc3f00b3a90 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.186+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f00a84b0 con 0x7fc3f00b3a90 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.187+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f0143160 con 0x7fc3f00b3a90 2023-09-29T23:06:14.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3de7fc700 1 -- 172.21.15.121:0/1260150529 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc3e8016bd0 con 0x7fc3f00b3a90 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3de7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3de7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3de7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3de7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3de7fc700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc3d0098780 0x7fc3d009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3ee59c700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc3d0098780 0x7fc3d009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.188+0000 7fc3de7fc700 1 -- 172.21.15.121:0/1260150529 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fc3e8012070 con 0x7fc3f00b3a90 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.189+0000 7fc3ee59c700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc3d0098780 0x7fc3d009ac30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40030c0 tx=0x7fc3e4015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.192+0000 7fc3de7fc700 1 -- 172.21.15.121:0/1260150529 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3e804e5a0 con 0x7fc3f00b3a90 2023-09-29T23:06:14.373 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.373+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc3f0150340 con 0x7fc3f00b3a90 2023-09-29T23:06:14.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.376+0000 7fc3de7fc700 1 -- 172.21.15.121:0/1260150529 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v14) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc3e808e430 con 0x7fc3f00b3a90 2023-09-29T23:06:14.376 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 8 2023-09-29T23:06:14.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 10 mgrc shutdown 2023-09-29T23:06:14.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc3d0098780 msgr2=0x7fc3d009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.378 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc3d0098780 0x7fc3d009ac30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40030c0 tx=0x7fc3e4015000 comp rx=0 tx=0).stop 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f00b3a90 msgr2=0x7fc3f0150680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f00b3a90 0x7fc3f0150680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800cef0 tx=0x7fc3e8007860 comp rx=0 tx=0).stop 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 shutdown_connections 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc3d0098780 0x7fc3d009ac30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3f00b3a90 0x7fc3f0150680 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3f006acb0 0x7fc3f00a9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 --2- 172.21.15.121:0/1260150529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006b080 0x7fc3f0142b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 >> 172.21.15.121:0/1260150529 conn(0x7fc3f006d160 msgr2=0x7fc3f00a4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:14.379 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.378+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 shutdown_connections 2023-09-29T23:06:14.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.379+0000 7fc3f4a88700 1 -- 172.21.15.121:0/1260150529 wait complete. 2023-09-29T23:06:14.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-29T23:06:14.509 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.509+0000 7f1da292b700 1 Processor -- start 2023-09-29T23:06:14.509 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.509+0000 7f1da292b700 1 -- start start 2023-09-29T23:06:14.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1da292b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d9c0630e0 0x7f1d9c142000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1da292b700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1d9c142540 0x7f1d9c09ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1da292b700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c142a60 con 0x7f1d9c0634b0 2023-09-29T23:06:14.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1da292b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c142c90 con 0x7f1d9c0630e0 2023-09-29T23:06:14.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1da292b700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c143f60 con 0x7f1d9c142540 2023-09-29T23:06:14.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d9c0630e0 0x7f1d9c142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d9c0630e0 0x7f1d9c142000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35126/0 (socket says 172.21.15.121:35126) 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9bfff700 1 -- 172.21.15.121:0/3798655628 learned_addr learned my addr 172.21.15.121:0/3798655628 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9b7fe700 1 --2- 172.21.15.121:0/3798655628 >> v2:172.21.15.123:3300/0 conn(0x7f1d9c142540 0x7f1d9c09ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9bfff700 1 -- 172.21.15.121:0/3798655628 >> v2:172.21.15.123:3300/0 conn(0x7f1d9c142540 msgr2=0x7f1d9c09ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9bfff700 1 --2- 172.21.15.121:0/3798655628 >> v2:172.21.15.123:3300/0 conn(0x7f1d9c142540 0x7f1d9c09ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9bfff700 1 -- 172.21.15.121:0/3798655628 >> v1:172.21.15.123:6789/0 conn(0x7f1d9c0634b0 legacy=0x7f1d9c09b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.510+0000 7f1d9bfff700 1 -- 172.21.15.121:0/3798655628 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c1440d0 con 0x7f1d9c0630e0 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1d9bfff700 1 --2- 172.21.15.121:0/3798655628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0630e0 0x7f1d9c142000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c09a580 tx=0x7f1d90009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599e7395fcf48de2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1d9affd700 1 -- 172.21.15.121:0/3798655628 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90005c10 con 0x7f1d9c0630e0 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1d9affd700 1 -- 172.21.15.121:0/3798655628 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d90005d70 con 0x7f1d9c0630e0 2023-09-29T23:06:14.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1d9affd700 1 -- 172.21.15.121:0/3798655628 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9000b840 con 0x7f1d9c0630e0 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 -- 172.21.15.121:0/3798655628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0630e0 msgr2=0x7f1d9c142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 --2- 172.21.15.121:0/3798655628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0630e0 0x7f1d9c142000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c09a580 tx=0x7f1d90009220 comp rx=0 tx=0).stop 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 -- 172.21.15.121:0/3798655628 shutdown_connections 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 --2- 172.21.15.121:0/3798655628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0630e0 0x7f1d9c142000 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 --2- 172.21.15.121:0/3798655628 >> v2:172.21.15.123:3300/0 conn(0x7f1d9c142540 0x7f1d9c09ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 -- 172.21.15.121:0/3798655628 >> 172.21.15.121:0/3798655628 conn(0x7f1d9c0a5bc0 msgr2=0x7f1d9c0b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 -- 172.21.15.121:0/3798655628 shutdown_connections 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.511+0000 7f1da292b700 1 -- 172.21.15.121:0/3798655628 wait complete. 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.512+0000 7f1da292b700 1 Processor -- start 2023-09-29T23:06:14.514 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.512+0000 7f1da292b700 1 -- start start 2023-09-29T23:06:14.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1da292b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9c0630e0 0x7f1d9c09f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1da292b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0634b0 0x7f1d9c135400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1da292b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d9c142540 0x7f1d9c1461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1da292b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c09ed40 con 0x7f1d9c0634b0 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1da292b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c135940 con 0x7f1d9c142540 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1da292b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c09d4e0 con 0x7f1d9c0630e0 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0634b0 0x7f1d9c135400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0634b0 0x7f1d9c135400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35130/0 (socket says 172.21.15.121:35130) 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 -- 172.21.15.121:0/1457453807 learned_addr learned my addr 172.21.15.121:0/1457453807 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9c0630e0 0x7f1d9c09f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1da0ec8700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d9c142540 0x7f1d9c1461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 -- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9c0630e0 msgr2=0x7f1d9c09f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9c0630e0 0x7f1d9c09f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 -- 172.21.15.121:0/1457453807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d9c142540 msgr2=0x7f1d9c1461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.517 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d9c142540 0x7f1d9c1461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 -- 172.21.15.121:0/1457453807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d90009480 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.513+0000 7f1d9b7fe700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0634b0 0x7f1d9c135400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00d8c0 tx=0x7f1d8c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.514+0000 7f1d83fff700 1 -- 172.21.15.121:0/1457453807 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0125f0 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.514+0000 7f1d83fff700 1 -- 172.21.15.121:0/1457453807 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c012750 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.514+0000 7f1d83fff700 1 -- 172.21.15.121:0/1457453807 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c012af0 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.514+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c134df0 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.514+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c06fa60 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.514+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c06abd0 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.515+0000 7f1d83fff700 1 -- 172.21.15.121:0/1457453807 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f1d8c01e460 con 0x7f1d9c0634b0 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.515+0000 7f1d83fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.516+0000 7f1d83fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:14.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.516+0000 7f1d83fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:14.519 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.516+0000 7f1d83fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:14.519 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.516+0000 7f1d83fff700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d88098780 0x7f1d8809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.519 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.516+0000 7f1d83fff700 1 -- 172.21.15.121:0/1457453807 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f1d8c0c05a0 con 0x7f1d9c0634b0 2023-09-29T23:06:14.519 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.520+0000 7f1d83fff700 1 -- 172.21.15.121:0/1457453807 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d8c08e030 con 0x7f1d9c0634b0 2023-09-29T23:06:14.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.527+0000 7f1d9bfff700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d88098780 0x7f1d8809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.527+0000 7f1d9bfff700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d88098780 0x7f1d8809ac30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1d90003080 tx=0x7f1d90015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.742+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d9c071660 con 0x7f1d9c0634b0 2023-09-29T23:06:14.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.742+0000 7f1d83fff700 1 -- 172.21.15.121:0/1457453807 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2774 (secure 0 0 0) 0x7f1d8c091650 con 0x7f1d9c0634b0 2023-09-29T23:06:14.742 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:14.743 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6811/3133589684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3133589684},{"type":"v1","addr":"172.21.15.121:6811","nonce":3133589684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6813/190961274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":190961274},{"type":"v1","addr":"172.21.15.121:6813","nonce":190961274}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6825/4268281148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":4268281148},{"type":"v1","addr":"172.21.15.123:6825","nonce":4268281148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-09-29T23:06:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 10 mgrc shutdown 2023-09-29T23:06:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d88098780 msgr2=0x7f1d8809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d88098780 0x7f1d8809ac30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1d90003080 tx=0x7f1d90015000 comp rx=0 tx=0).stop 2023-09-29T23:06:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0634b0 msgr2=0x7f1d9c135400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0634b0 0x7f1d9c135400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00d8c0 tx=0x7f1d8c008040 comp rx=0 tx=0).stop 2023-09-29T23:06:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 shutdown_connections 2023-09-29T23:06:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d88098780 0x7f1d8809ac30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d9c142540 0x7f1d9c1461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9c0630e0 0x7f1d9c09f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 --2- 172.21.15.121:0/1457453807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9c0634b0 0x7f1d9c135400 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 >> 172.21.15.121:0/1457453807 conn(0x7f1d9c0a5bc0 msgr2=0x7f1d9c069970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 shutdown_connections 2023-09-29T23:06:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.744+0000 7f1da292b700 1 -- 172.21.15.121:0/1457453807 wait complete. 2023-09-29T23:06:14.747 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 3 2023-09-29T23:06:14.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.873+0000 7ff25cfb3700 1 Processor -- start 2023-09-29T23:06:14.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.873+0000 7ff25cfb3700 1 -- start start 2023-09-29T23:06:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.873+0000 7ff25cfb3700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff25806f960 0x7ff258140c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.873+0000 7ff25cfb3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258141170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.873+0000 7ff25cfb3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff258143d00 con 0x7ff2580b4ba0 2023-09-29T23:06:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.873+0000 7ff25cfb3700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff258143fc0 con 0x7ff25806f960 2023-09-29T23:06:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.873+0000 7ff25cfb3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b4f70 con 0x7ff2580a5440 2023-09-29T23:06:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50180/0 (socket says 172.21.15.121:50180) 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 -- 172.21.15.121:0/3721515794 learned_addr learned my addr 172.21.15.121:0/3721515794 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff255d9b700 1 --2- 172.21.15.121:0/3721515794 >> v2:172.21.15.123:3300/0 conn(0x7ff25806f960 0x7ff258140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 -- 172.21.15.121:0/3721515794 >> v2:172.21.15.123:3300/0 conn(0x7ff25806f960 msgr2=0x7ff258140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 --2- 172.21.15.121:0/3721515794 >> v2:172.21.15.123:3300/0 conn(0x7ff25806f960 0x7ff258140c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 -- 172.21.15.121:0/3721515794 >> v1:172.21.15.123:6789/0 conn(0x7ff2580b4ba0 legacy=0x7ff2581444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 -- 172.21.15.121:0/3721515794 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2581405d0 con 0x7ff2580a5440 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff256d9d700 1 --2- 172.21.15.121:0/3721515794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258141170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff250009f20 tx=0x7ff250009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=457205de6bf7dba6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff25559a700 1 -- 172.21.15.121:0/3721515794 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25000e070 con 0x7ff2580a5440 2023-09-29T23:06:14.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff25559a700 1 -- 172.21.15.121:0/3721515794 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25000b9d0 con 0x7ff2580a5440 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.874+0000 7ff25559a700 1 -- 172.21.15.121:0/3721515794 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500092e0 con 0x7ff2580a5440 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 -- 172.21.15.121:0/3721515794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 msgr2=0x7ff258141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/3721515794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258141170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff250009f20 tx=0x7ff250009840 comp rx=0 tx=0).stop 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 -- 172.21.15.121:0/3721515794 shutdown_connections 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/3721515794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258141170 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/3721515794 >> v2:172.21.15.123:3300/0 conn(0x7ff25806f960 0x7ff258140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 -- 172.21.15.121:0/3721515794 >> 172.21.15.121:0/3721515794 conn(0x7ff258067440 msgr2=0x7ff258067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 -- 172.21.15.121:0/3721515794 shutdown_connections 2023-09-29T23:06:14.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.875+0000 7ff25cfb3700 1 -- 172.21.15.121:0/3721515794 wait complete. 2023-09-29T23:06:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.876+0000 7ff25cfb3700 1 Processor -- start 2023-09-29T23:06:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.876+0000 7ff25cfb3700 1 -- start start 2023-09-29T23:06:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25cfb3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff25806f960 0x7ff25809ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25cfb3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25cfb3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2580b4ba0 0x7ff258137170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25cfb3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580a4550 con 0x7ff2580b4ba0 2023-09-29T23:06:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25cfb3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25809f560 con 0x7ff25806f960 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25cfb3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff258151420 con 0x7ff2580a5440 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff256d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2580b4ba0 0x7ff258137170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff25806f960 0x7ff25809ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50190/0 (socket says 172.21.15.121:50190) 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff256d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2580b4ba0 0x7ff258137170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35134/0 (socket says 172.21.15.121:35134) 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff25659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff25806f960 0x7ff25809ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.121:48944/0 (socket says 172.21.15.121:48944) 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 -- 172.21.15.121:0/574701765 learned_addr learned my addr 172.21.15.121:0/574701765 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 -- 172.21.15.121:0/574701765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff25806f960 msgr2=0x7ff25809ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff25806f960 0x7ff25809ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 -- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2580b4ba0 msgr2=0x7ff258137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2580b4ba0 0x7ff258137170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 -- 172.21.15.121:0/574701765 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250009ae0 con 0x7ff2580a5440 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff256d9d700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2580b4ba0 0x7ff258137170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff255d9b700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258142a50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00e900 tx=0x7ff24c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.877+0000 7ff2467fc700 1 -- 172.21.15.121:0/574701765 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c013680 con 0x7ff2580a5440 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.878+0000 7ff2467fc700 1 -- 172.21.15.121:0/574701765 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c0137e0 con 0x7ff2580a5440 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.878+0000 7ff2467fc700 1 -- 172.21.15.121:0/574701765 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c013ba0 con 0x7ff2580a5440 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.878+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff258143890 con 0x7ff2580a5440 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.878+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2581376b0 con 0x7ff2580a5440 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.878+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2580a64b0 con 0x7ff2580a5440 2023-09-29T23:06:14.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.879+0000 7ff2467fc700 1 -- 172.21.15.121:0/574701765 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff24c0253d0 con 0x7ff2580a5440 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.879+0000 7ff2467fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.880+0000 7ff2467fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.880+0000 7ff2467fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.880+0000 7ff2467fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.880+0000 7ff2467fc700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff238098780 0x7ff23809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.880+0000 7ff25659c700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff238098780 0x7ff23809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.880+0000 7ff2467fc700 1 -- 172.21.15.121:0/574701765 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7ff24c0c08f0 con 0x7ff2580a5440 2023-09-29T23:06:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.880+0000 7ff25659c700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff238098780 0x7ff23809ac30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff248003a80 tx=0x7ff248007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:14.884 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:14.883+0000 7ff2467fc700 1 -- 172.21.15.121:0/574701765 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff24c08e380 con 0x7ff2580a5440 2023-09-29T23:06:15.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.063+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff258137930 con 0x7ff2580a5440 2023-09-29T23:06:15.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.064+0000 7ff2467fc700 1 -- 172.21.15.121:0/574701765 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+9037 (secure 0 0 0) 0x7ff24c017090 con 0x7ff2580a5440 2023-09-29T23:06:15.064 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:15.064 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":15,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:14.373274+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:15.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 10 mgrc shutdown 2023-09-29T23:06:15.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff238098780 msgr2=0x7ff23809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff238098780 0x7ff23809ac30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff248003a80 tx=0x7ff248007ec0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 msgr2=0x7ff258142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258142a50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00e900 tx=0x7ff24c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 shutdown_connections 2023-09-29T23:06:15.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff238098780 0x7ff23809ac30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2580b4ba0 0x7ff258137170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff25806f960 0x7ff25809ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 --2- 172.21.15.121:0/574701765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2580a5440 0x7ff258142a50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 >> 172.21.15.121:0/574701765 conn(0x7ff258067440 msgr2=0x7ff2580a6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:15.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 shutdown_connections 2023-09-29T23:06:15.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.066+0000 7ff25cfb3700 1 -- 172.21.15.121:0/574701765 wait complete. 2023-09-29T23:06:15.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-29T23:06:07.527057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-09-29T23:06:12.101557+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-09-29T23:06:15.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-29T23:06:15.197 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.197+0000 7faa3b7ad700 1 Processor -- start 2023-09-29T23:06:15.197 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.197+0000 7faa3b7ad700 1 -- start start 2023-09-29T23:06:15.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa3b7ad700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faa340a0040 0x7faa3406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa3b7ad700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa34141b20 0x7faa34141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa3b7ad700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa340b1ec0 con 0x7faa340b1af0 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa3b7ad700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa341424e0 con 0x7faa340a0040 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa3b7ad700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa34134410 con 0x7faa34141b20 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa34141b20 0x7faa34141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa34141b20 0x7faa34141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50194/0 (socket says 172.21.15.121:50194) 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 -- 172.21.15.121:0/3243558270 learned_addr learned my addr 172.21.15.121:0/3243558270 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa38d48700 1 --2- 172.21.15.121:0/3243558270 >> v2:172.21.15.123:3300/0 conn(0x7faa340a0040 0x7faa3406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 -- 172.21.15.121:0/3243558270 >> v2:172.21.15.123:3300/0 conn(0x7faa340a0040 msgr2=0x7faa3406f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 --2- 172.21.15.121:0/3243558270 >> v2:172.21.15.123:3300/0 conn(0x7faa340a0040 0x7faa3406f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 -- 172.21.15.121:0/3243558270 >> v1:172.21.15.123:6789/0 conn(0x7faa340b1af0 legacy=0x7faa3409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:15.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 -- 172.21.15.121:0/3243558270 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3409f3d0 con 0x7faa34141b20 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa39d4a700 1 --2- 172.21.15.121:0/3243558270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa34141b20 0x7faa34141fa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009f20 tx=0x7faa2c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ddbeeeb2cd5f2db server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa2bfff700 1 -- 172.21.15.121:0/3243558270 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c00e070 con 0x7faa34141b20 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa2bfff700 1 -- 172.21.15.121:0/3243558270 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2c00ba40 con 0x7faa34141b20 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.198+0000 7faa2bfff700 1 -- 172.21.15.121:0/3243558270 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2c0092e0 con 0x7faa34141b20 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3243558270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa34141b20 msgr2=0x7faa34141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3243558270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa34141b20 0x7faa34141fa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009f20 tx=0x7faa2c009840 comp rx=0 tx=0).stop 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3243558270 shutdown_connections 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3243558270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa34141b20 0x7faa34141fa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3243558270 >> v2:172.21.15.123:3300/0 conn(0x7faa340a0040 0x7faa3406f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3243558270 >> 172.21.15.121:0/3243558270 conn(0x7faa34069cd0 msgr2=0x7faa3406a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:15.201 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3243558270 shutdown_connections 2023-09-29T23:06:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.199+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3243558270 wait complete. 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.200+0000 7faa3b7ad700 1 Processor -- start 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.200+0000 7faa3b7ad700 1 -- start start 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa3b7ad700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa340a0040 0x7faa340a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa3b7ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa340b1af0 0x7faa3413e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa3b7ad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34141b20 0x7faa3413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa3b7ad700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa340a5370 con 0x7faa340b1af0 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa3b7ad700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3413ed50 con 0x7faa34141b20 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa3b7ad700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3413eff0 con 0x7faa340a0040 2023-09-29T23:06:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa340a0040 0x7faa340a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa340a0040 0x7faa340a2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50196/0 (socket says 172.21.15.121:50196) 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa38d48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa340b1af0 0x7faa3413e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 -- 172.21.15.121:0/3643617148 learned_addr learned my addr 172.21.15.121:0/3643617148 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39d4a700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34141b20 0x7faa3413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 -- 172.21.15.121:0/3643617148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34141b20 msgr2=0x7faa3413cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34141b20 0x7faa3413cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 -- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa340b1af0 msgr2=0x7faa3413e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa340b1af0 0x7faa3413e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 -- 172.21.15.121:0/3643617148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c009ae0 con 0x7faa340a0040 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa38d48700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa340b1af0 0x7faa3413e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:15.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.201+0000 7faa39549700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa340a0040 0x7faa340a2480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c5b0 tx=0x7faa240020e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.202+0000 7faa297fa700 1 -- 172.21.15.121:0/3643617148 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24011590 con 0x7faa340a0040 2023-09-29T23:06:15.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.202+0000 7faa297fa700 1 -- 172.21.15.121:0/3643617148 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa240116f0 con 0x7faa340a0040 2023-09-29T23:06:15.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.202+0000 7faa297fa700 1 -- 172.21.15.121:0/3643617148 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24011a90 con 0x7faa340a0040 2023-09-29T23:06:15.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.202+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3413f290 con 0x7faa340a0040 2023-09-29T23:06:15.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.202+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa340a5840 con 0x7faa340a0040 2023-09-29T23:06:15.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.203+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3413e180 con 0x7faa340a0040 2023-09-29T23:06:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.204+0000 7faa297fa700 1 -- 172.21.15.121:0/3643617148 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7faa2401c460 con 0x7faa340a0040 2023-09-29T23:06:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.204+0000 7faa297fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.205+0000 7faa297fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.205+0000 7faa297fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.205+0000 7faa297fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.205+0000 7faa297fa700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7faa18098780 0x7faa1809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.205+0000 7faa38d48700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7faa18098780 0x7faa1809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.205+0000 7faa297fa700 1 -- 172.21.15.121:0/3643617148 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7faa240c08c0 con 0x7faa340a0040 2023-09-29T23:06:15.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.205+0000 7faa38d48700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7faa18098780 0x7faa1809ac30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faa30006fd0 tx=0x7faa30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.208+0000 7faa297fa700 1 -- 172.21.15.121:0/3643617148 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa2408e220 con 0x7faa340a0040 2023-09-29T23:06:15.387 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.387+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa34071660 con 0x7faa340a0040 2023-09-29T23:06:15.387 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.388+0000 7faa297fa700 1 -- 172.21.15.121:0/3643617148 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa24091840 con 0x7faa340a0040 2023-09-29T23:06:15.388 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2023-09-29T23:06:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 10 mgrc shutdown 2023-09-29T23:06:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7faa18098780 msgr2=0x7faa1809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7faa18098780 0x7faa1809ac30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faa30006fd0 tx=0x7faa30008040 comp rx=0 tx=0).stop 2023-09-29T23:06:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa340a0040 msgr2=0x7faa340a2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa340a0040 0x7faa340a2480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c5b0 tx=0x7faa240020e0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 shutdown_connections 2023-09-29T23:06:15.390 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7faa18098780 0x7faa1809ac30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34141b20 0x7faa3413cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faa340a0040 0x7faa340a2480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 --2- 172.21.15.121:0/3643617148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa340b1af0 0x7faa3413e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 >> 172.21.15.121:0/3643617148 conn(0x7faa34069cd0 msgr2=0x7faa340b0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 shutdown_connections 2023-09-29T23:06:15.391 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.390+0000 7faa3b7ad700 1 -- 172.21.15.121:0/3643617148 wait complete. 2023-09-29T23:06:15.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-09-29T23:06:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.520+0000 7f7a27882700 1 Processor -- start 2023-09-29T23:06:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a27882700 1 -- start start 2023-09-29T23:06:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a27882700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a200b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a27882700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2009a810 con 0x7f7a200632a0 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a27882700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200a4ab0 con 0x7f7a200a5550 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a27882700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200a4d50 con 0x7f7a200a4570 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a24e1d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a200b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a2561e700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7a200632a0 0x7f7a2009b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45038/0 (socket says 172.21.15.121:45038) 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a2561e700 1 -- 172.21.15.121:0/943775061 learned_addr learned my addr 172.21.15.121:0/943775061 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a24e1d700 1 -- 172.21.15.121:0/943775061 >> v1:172.21.15.123:6789/0 conn(0x7f7a200a5550 legacy=0x7f7a2014b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.521+0000 7f7a17fff700 1 -- 172.21.15.121:0/943775061 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981445147 0 0) 0x7f7a2009a810 con 0x7f7a200632a0 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a24e1d700 1 -- 172.21.15.121:0/943775061 >> v1:172.21.15.121:6789/0 conn(0x7f7a200632a0 legacy=0x7f7a2009b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a24e1d700 1 -- 172.21.15.121:0/943775061 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2009af80 con 0x7f7a200a4570 2023-09-29T23:06:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a24e1d700 1 --2- 172.21.15.121:0/943775061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a200b38f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009910 tx=0x7f7a1c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc8dab847fbb3066 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a17fff700 1 -- 172.21.15.121:0/943775061 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00b860 con 0x7f7a200a4570 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a17fff700 1 -- 172.21.15.121:0/943775061 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c00b9c0 con 0x7f7a200a4570 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a17fff700 1 -- 172.21.15.121:0/943775061 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00bd60 con 0x7f7a200a4570 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a27882700 1 -- 172.21.15.121:0/943775061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 msgr2=0x7f7a200b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a27882700 1 --2- 172.21.15.121:0/943775061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a200b38f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009910 tx=0x7f7a1c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a27882700 1 -- 172.21.15.121:0/943775061 shutdown_connections 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a27882700 1 --2- 172.21.15.121:0/943775061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a200b38f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a27882700 1 -- 172.21.15.121:0/943775061 >> 172.21.15.121:0/943775061 conn(0x7f7a20069ec0 msgr2=0x7f7a200b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a27882700 1 -- 172.21.15.121:0/943775061 shutdown_connections 2023-09-29T23:06:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.522+0000 7f7a27882700 1 -- 172.21.15.121:0/943775061 wait complete. 2023-09-29T23:06:15.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.523+0000 7f7a27882700 1 Processor -- start 2023-09-29T23:06:15.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.523+0000 7f7a27882700 1 -- start start 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a27882700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a200632a0 0x7f7a20143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a27882700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a2009cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a27882700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a200a5550 0x7f7a2006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a27882700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2013eaa0 con 0x7f7a200632a0 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a27882700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2009c710 con 0x7f7a200a5550 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a27882700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2006f820 con 0x7f7a200a4570 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a2009cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a2561e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a200632a0 0x7f7a20143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a2009cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50202/0 (socket says 172.21.15.121:50202) 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 -- 172.21.15.121:0/3200800585 learned_addr learned my addr 172.21.15.121:0/3200800585 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:15.527 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a25e1f700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a200a5550 0x7f7a2006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 -- 172.21.15.121:0/3200800585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a200a5550 msgr2=0x7f7a2006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a200a5550 0x7f7a2006e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 -- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a200632a0 msgr2=0x7f7a20143990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a200632a0 0x7f7a20143990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a24e1d700 1 -- 172.21.15.121:0/3200800585 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c009420 con 0x7f7a200a4570 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.524+0000 7f7a2561e700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a200632a0 0x7f7a20143990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.525+0000 7f7a24e1d700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a2009cf70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a2006c400 tx=0x7f7a1c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.525+0000 7f7a157fa700 1 -- 172.21.15.121:0/3200800585 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c01f070 con 0x7f7a200a4570 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.525+0000 7f7a157fa700 1 -- 172.21.15.121:0/3200800585 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c002f70 con 0x7f7a200a4570 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.525+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2009e280 con 0x7f7a200a4570 2023-09-29T23:06:15.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.525+0000 7f7a157fa700 1 -- 172.21.15.121:0/3200800585 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c025420 con 0x7f7a200a4570 2023-09-29T23:06:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.525+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a20144040 con 0x7f7a200a4570 2023-09-29T23:06:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.526+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a200b3960 con 0x7f7a200a4570 2023-09-29T23:06:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a157fa700 1 -- 172.21.15.121:0/3200800585 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7a1c023070 con 0x7f7a200a4570 2023-09-29T23:06:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a157fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a157fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a157fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a157fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a157fa700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7a04098780 0x7f7a0409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a157fa700 1 -- 172.21.15.121:0/3200800585 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f7a1c0c1910 con 0x7f7a200a4570 2023-09-29T23:06:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.527+0000 7f7a2561e700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7a04098780 0x7f7a0409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.528+0000 7f7a2561e700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7a04098780 0x7f7a0409ac30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a10007fd0 tx=0x7f7a10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.530+0000 7f7a157fa700 1 -- 172.21.15.121:0/3200800585 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a1c08f3a0 con 0x7f7a200a4570 2023-09-29T23:06:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.711+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7a2006ab80 con 0x7f7a200a4570 2023-09-29T23:06:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.712+0000 7f7a157fa700 1 -- 172.21.15.121:0/3200800585 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7a1c02a0a0 con 0x7f7a200a4570 2023-09-29T23:06:15.712 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 8 2023-09-29T23:06:15.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 10 mgrc shutdown 2023-09-29T23:06:15.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7a04098780 msgr2=0x7f7a0409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7a04098780 0x7f7a0409ac30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a10007fd0 tx=0x7f7a10009040 comp rx=0 tx=0).stop 2023-09-29T23:06:15.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 msgr2=0x7f7a2009cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a2009cf70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a2006c400 tx=0x7f7a1c017000 comp rx=0 tx=0).stop 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 shutdown_connections 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7a04098780 0x7f7a0409ac30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a200a5550 0x7f7a2006e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a200632a0 0x7f7a20143990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 --2- 172.21.15.121:0/3200800585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200a4570 0x7f7a2009cf70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 >> 172.21.15.121:0/3200800585 conn(0x7f7a20069ec0 msgr2=0x7f7a20070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 shutdown_connections 2023-09-29T23:06:15.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.714+0000 7f7a27882700 1 -- 172.21.15.121:0/3200800585 wait complete. 2023-09-29T23:06:15.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-09-29T23:06:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.842+0000 7fb433d64700 1 Processor -- start 2023-09-29T23:06:15.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.842+0000 7fb433d64700 1 -- start start 2023-09-29T23:06:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb433d64700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb42c142560 0x7fb42c0b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb433d64700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb42c0a5c80 0x7fb42c0a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb433d64700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c142930 con 0x7fb42c0a6080 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb433d64700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c13e380 con 0x7fb42c0a5c80 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb433d64700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c0a5990 con 0x7fb42c142560 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb4312ff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb42c0a5c80 0x7fb42c0a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb432301700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb42c0a6080 0x7fb42c06dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45054/0 (socket says 172.21.15.121:45054) 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb432301700 1 -- 172.21.15.121:0/2611530974 learned_addr learned my addr 172.21.15.121:0/2611530974 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb431b00700 1 --2- 172.21.15.121:0/2611530974 >> v2:172.21.15.123:3300/0 conn(0x7fb42c142560 0x7fb42c0b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb4312ff700 1 -- 172.21.15.121:0/2611530974 >> v2:172.21.15.123:3300/0 conn(0x7fb42c142560 msgr2=0x7fb42c0b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb4312ff700 1 --2- 172.21.15.121:0/2611530974 >> v2:172.21.15.123:3300/0 conn(0x7fb42c142560 0x7fb42c0b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb4312ff700 1 -- 172.21.15.121:0/2611530974 >> v1:172.21.15.121:6789/0 conn(0x7fb42c0a6080 legacy=0x7fb42c06dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb430afe700 1 -- 172.21.15.121:0/2611530974 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674322752 0 0) 0x7fb42c142930 con 0x7fb42c0a6080 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb4312ff700 1 -- 172.21.15.121:0/2611530974 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c141af0 con 0x7fb42c0a5c80 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.843+0000 7fb4312ff700 1 --2- 172.21.15.121:0/2611530974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a4f30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0681f0 tx=0x7fb4280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d2aa93c4327ab4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb430afe700 1 -- 172.21.15.121:0/2611530974 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42800b6b0 con 0x7fb42c0a5c80 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb430afe700 1 -- 172.21.15.121:0/2611530974 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42800b810 con 0x7fb42c0a5c80 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb430afe700 1 -- 172.21.15.121:0/2611530974 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42800bbb0 con 0x7fb42c0a5c80 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 -- 172.21.15.121:0/2611530974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 msgr2=0x7fb42c0a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 --2- 172.21.15.121:0/2611530974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a4f30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb42c0681f0 tx=0x7fb4280091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 -- 172.21.15.121:0/2611530974 shutdown_connections 2023-09-29T23:06:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 --2- 172.21.15.121:0/2611530974 >> v2:172.21.15.123:3300/0 conn(0x7fb42c142560 0x7fb42c0b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 --2- 172.21.15.121:0/2611530974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a4f30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 -- 172.21.15.121:0/2611530974 >> 172.21.15.121:0/2611530974 conn(0x7fb42c067440 msgr2=0x7fb42c067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 -- 172.21.15.121:0/2611530974 shutdown_connections 2023-09-29T23:06:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.844+0000 7fb433d64700 1 -- 172.21.15.121:0/2611530974 wait complete. 2023-09-29T23:06:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.845+0000 7fb433d64700 1 Processor -- start 2023-09-29T23:06:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.845+0000 7fb433d64700 1 -- start start 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb433d64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb433d64700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42c142560 0x7fb42c098040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb433d64700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c098580 0x7fb42c096a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb433d64700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c0a2740 con 0x7fb42c0a5c80 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb433d64700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c0a2920 con 0x7fb42c098580 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb433d64700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c0a3b20 con 0x7fb42c142560 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42c142560 0x7fb42c098040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42c142560 0x7fb42c098040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50218/0 (socket says 172.21.15.121:50218) 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 -- 172.21.15.121:0/1497867937 learned_addr learned my addr 172.21.15.121:0/1497867937 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb431b00700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb432301700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c098580 0x7fb42c096a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 -- 172.21.15.121:0/1497867937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c098580 msgr2=0x7fb42c096a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c098580 0x7fb42c096a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 -- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 msgr2=0x7fb42c0a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 -- 172.21.15.121:0/1497867937 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428009420 con 0x7fb42c142560 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb431b00700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb4312ff700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42c142560 0x7fb42c098040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb4280091e0 tx=0x7fb428017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb421ffb700 1 -- 172.21.15.121:0/1497867937 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42800b980 con 0x7fb42c142560 2023-09-29T23:06:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb421ffb700 1 -- 172.21.15.121:0/1497867937 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42800bae0 con 0x7fb42c142560 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c0a4690 con 0x7fb42c142560 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.846+0000 7fb421ffb700 1 -- 172.21.15.121:0/1497867937 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428011670 con 0x7fb42c142560 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.847+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c0a2260 con 0x7fb42c142560 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.847+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c071660 con 0x7fb42c142560 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb421ffb700 1 -- 172.21.15.121:0/1497867937 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb428011aa0 con 0x7fb42c142560 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb421ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb421ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb421ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb421ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb421ffb700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb410098700 0x7fb41009abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb421ffb700 1 -- 172.21.15.121:0/1497867937 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fb4280c21f0 con 0x7fb42c142560 2023-09-29T23:06:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.849+0000 7fb431b00700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb410098700 0x7fb41009abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.850+0000 7fb431b00700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb410098700 0x7fb41009abb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006fd0 tx=0x7fb41c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:15.853+0000 7fb421ffb700 1 -- 172.21.15.121:0/1497867937 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb42808fc80 con 0x7fb42c142560 2023-09-29T23:06:16.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.036+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb42c0b43f0 con 0x7fb42c142560 2023-09-29T23:06:16.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.037+0000 7fb421ffb700 1 -- 172.21.15.121:0/1497867937 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v15) v1 ==== 84+0+126 (secure 0 0 0) 0x7fb428033090 con 0x7fb42c142560 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout:[ 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "poolnum": 1, 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "poolname": ".mgr" 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "poolnum": 2, 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "poolname": "rbd" 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-09-29T23:06:16.037 INFO:teuthology.orchestra.run.smithi121.stdout:] 2023-09-29T23:06:16.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.038+0000 7fb433d64700 10 mgrc shutdown 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.038+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb410098700 msgr2=0x7fb41009abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.038+0000 7fb433d64700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb410098700 0x7fb41009abb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006fd0 tx=0x7fb41c008040 comp rx=0 tx=0).stop 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.038+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42c142560 msgr2=0x7fb42c098040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.038+0000 7fb433d64700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42c142560 0x7fb42c098040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb4280091e0 tx=0x7fb428017000 comp rx=0 tx=0).stop 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 shutdown_connections 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb410098700 0x7fb41009abb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb42c098580 0x7fb42c096a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42c0a5c80 0x7fb42c0a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 --2- 172.21.15.121:0/1497867937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42c142560 0x7fb42c098040 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 >> 172.21.15.121:0/1497867937 conn(0x7fb42c067440 msgr2=0x7fb42c13ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:16.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 shutdown_connections 2023-09-29T23:06:16.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.039+0000 7fb433d64700 1 -- 172.21.15.121:0/1497867937 wait complete. 2023-09-29T23:06:16.050 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-09-29T23:06:16.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-09-29T23:06:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.171+0000 7efe1d16d700 1 Processor -- start 2023-09-29T23:06:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.171+0000 7efe1d16d700 1 -- start start 2023-09-29T23:06:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1d16d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe18063330 0x7efe180a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1d16d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe180a3640 0x7efe181438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1d16d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe18063700 con 0x7efe1813f880 2023-09-29T23:06:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1d16d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe18144170 con 0x7efe180a3640 2023-09-29T23:06:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1d16d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe18144430 con 0x7efe18063330 2023-09-29T23:06:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe16d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe18063330 0x7efe180a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1759e700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efe1813f880 0x7efe18099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45070/0 (socket says 172.21.15.121:45070) 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1759e700 1 -- 172.21.15.121:0/3856263107 learned_addr learned my addr 172.21.15.121:0/3856263107 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe1659c700 1 --2- 172.21.15.121:0/3856263107 >> v2:172.21.15.123:3300/0 conn(0x7efe180a3640 0x7efe181438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe16d9d700 1 -- 172.21.15.121:0/3856263107 >> v2:172.21.15.123:3300/0 conn(0x7efe180a3640 msgr2=0x7efe181438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe16d9d700 1 --2- 172.21.15.121:0/3856263107 >> v2:172.21.15.123:3300/0 conn(0x7efe180a3640 0x7efe181438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe15d9b700 1 -- 172.21.15.121:0/3856263107 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224201595 0 0) 0x7efe18063700 con 0x7efe1813f880 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe16d9d700 1 -- 172.21.15.121:0/3856263107 >> v1:172.21.15.121:6789/0 conn(0x7efe1813f880 legacy=0x7efe18099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.172+0000 7efe16d9d700 1 -- 172.21.15.121:0/3856263107 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe180a3a10 con 0x7efe18063330 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe16d9d700 1 --2- 172.21.15.121:0/3856263107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe18063330 0x7efe180a47e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe000098e0 tx=0x7efe000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=834f4e928a240015 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe15d9b700 1 -- 172.21.15.121:0/3856263107 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe00009e80 con 0x7efe18063330 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe15d9b700 1 -- 172.21.15.121:0/3856263107 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0000b860 con 0x7efe18063330 2023-09-29T23:06:16.175 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe15d9b700 1 -- 172.21.15.121:0/3856263107 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0000bbe0 con 0x7efe18063330 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 -- 172.21.15.121:0/3856263107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe18063330 msgr2=0x7efe180a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 --2- 172.21.15.121:0/3856263107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe18063330 0x7efe180a47e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe000098e0 tx=0x7efe000091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 -- 172.21.15.121:0/3856263107 shutdown_connections 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 --2- 172.21.15.121:0/3856263107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe18063330 0x7efe180a47e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 --2- 172.21.15.121:0/3856263107 >> v2:172.21.15.123:3300/0 conn(0x7efe180a3640 0x7efe181438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 -- 172.21.15.121:0/3856263107 >> 172.21.15.121:0/3856263107 conn(0x7efe180b3b90 msgr2=0x7efe18067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 -- 172.21.15.121:0/3856263107 shutdown_connections 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.173+0000 7efe1d16d700 1 -- 172.21.15.121:0/3856263107 wait complete. 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.174+0000 7efe1d16d700 1 Processor -- start 2023-09-29T23:06:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.174+0000 7efe1d16d700 1 -- start start 2023-09-29T23:06:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1d16d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe18063330 0x7efe1813e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1d16d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe180a3640 0x7efe1814fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1d16d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe1813f880 0x7efe18150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1d16d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe18150e40 con 0x7efe1813f880 2023-09-29T23:06:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1d16d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe18150970 con 0x7efe18063330 2023-09-29T23:06:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1d16d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1809e300 con 0x7efe180a3640 2023-09-29T23:06:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe180a3640 0x7efe1814fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe180a3640 0x7efe1814fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50242/0 (socket says 172.21.15.121:50242) 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe1813f880 0x7efe18150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 -- 172.21.15.121:0/2505552771 learned_addr learned my addr 172.21.15.121:0/2505552771 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe16d9d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe18063330 0x7efe1813e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 -- 172.21.15.121:0/2505552771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe18063330 msgr2=0x7efe1813e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe18063330 0x7efe1813e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 -- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe1813f880 msgr2=0x7efe18150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe1813f880 0x7efe18150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 -- 172.21.15.121:0/2505552771 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe00009420 con 0x7efe180a3640 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1759e700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe1813f880 0x7efe18150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.175+0000 7efe1659c700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe180a3640 0x7efe1814fef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe0800d8c0 tx=0x7efe08008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.176+0000 7efe0effd700 1 -- 172.21.15.121:0/2505552771 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe080125f0 con 0x7efe180a3640 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.176+0000 7efe0effd700 1 -- 172.21.15.121:0/2505552771 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe08012750 con 0x7efe180a3640 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.176+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe18150fe0 con 0x7efe180a3640 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.176+0000 7efe0effd700 1 -- 172.21.15.121:0/2505552771 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08012af0 con 0x7efe180a3640 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.176+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1809e020 con 0x7efe180a3640 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.176+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe18064c00 con 0x7efe180a3640 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.177+0000 7efe0effd700 1 -- 172.21.15.121:0/2505552771 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7efe0801e460 con 0x7efe180a3640 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.177+0000 7efe0effd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.178+0000 7efe0effd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.178+0000 7efe0effd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.178+0000 7efe0effd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:16.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.178+0000 7efe0effd700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efdf80987b0 0x7efdf809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:16.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.178+0000 7efe16d9d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efdf80987b0 0x7efdf809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:16.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.178+0000 7efe0effd700 1 -- 172.21.15.121:0/2505552771 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7efe080c0620 con 0x7efe180a3640 2023-09-29T23:06:16.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.178+0000 7efe16d9d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efdf80987b0 0x7efdf809ac60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe000091e0 tx=0x7efe00017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:16.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.182+0000 7efe0effd700 1 -- 172.21.15.121:0/2505552771 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe08091b40 con 0x7efe180a3640 2023-09-29T23:06:16.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.364+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efe18143df0 con 0x7efe180a3640 2023-09-29T23:06:16.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.866+0000 7efe0effd700 1 -- 172.21.15.121:0/2505552771 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v16) v1 ==== 138+0+0 (secure 0 0 0) 0x7efe080916d0 con 0x7efe180a3640 2023-09-29T23:06:16.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 10 mgrc shutdown 2023-09-29T23:06:16.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efdf80987b0 msgr2=0x7efdf809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efdf80987b0 0x7efdf809ac60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe000091e0 tx=0x7efe00017000 comp rx=0 tx=0).stop 2023-09-29T23:06:16.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe180a3640 msgr2=0x7efe1814fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe180a3640 0x7efe1814fef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe0800d8c0 tx=0x7efe08008040 comp rx=0 tx=0).stop 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 shutdown_connections 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efdf80987b0 0x7efdf809ac60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe1813f880 0x7efe18150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe18063330 0x7efe1813e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 --2- 172.21.15.121:0/2505552771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe180a3640 0x7efe1814fef0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 >> 172.21.15.121:0/2505552771 conn(0x7efe180b3b90 msgr2=0x7efe18143650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 shutdown_connections 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:16.869+0000 7efe1d16d700 1 -- 172.21.15.121:0/2505552771 wait complete. 2023-09-29T23:06:16.878 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_metadata' created 2023-09-29T23:06:16.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-09-29T23:06:17.008 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.008+0000 7fb74b1bd700 1 Processor -- start 2023-09-29T23:06:17.008 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb74b1bd700 1 -- start start 2023-09-29T23:06:17.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb74b1bd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb74409f290 0x7fb7441409e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:17.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb74b1bd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409db40 0x7fb744147d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:17.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb74b1bd700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb744063700 con 0x7fb744063330 2023-09-29T23:06:17.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb74b1bd700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74409f6d0 con 0x7fb74409f290 2023-09-29T23:06:17.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb74b1bd700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74409dfc0 con 0x7fb74409db40 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb74975a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409db40 0x7fb744147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb748f59700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb744063330 0x7fb74413ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53882/0 (socket says 172.21.15.121:53882) 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb748f59700 1 -- 172.21.15.121:0/2408997027 learned_addr learned my addr 172.21.15.121:0/2408997027 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb743fff700 1 --2- 172.21.15.121:0/2408997027 >> v2:172.21.15.123:3300/0 conn(0x7fb74409f290 0x7fb7441409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.009+0000 7fb7437fe700 1 -- 172.21.15.121:0/2408997027 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003717879 0 0) 0x7fb744063700 con 0x7fb744063330 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb7437fe700 1 -- 172.21.15.121:0/2408997027 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb734003570 con 0x7fb744063330 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74975a700 1 -- 172.21.15.121:0/2408997027 >> v2:172.21.15.123:3300/0 conn(0x7fb74409f290 msgr2=0x7fb7441409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74975a700 1 --2- 172.21.15.121:0/2408997027 >> v2:172.21.15.123:3300/0 conn(0x7fb74409f290 0x7fb7441409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74975a700 1 -- 172.21.15.121:0/2408997027 >> v1:172.21.15.121:6789/0 conn(0x7fb744063330 legacy=0x7fb74413ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74975a700 1 -- 172.21.15.121:0/2408997027 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744141050 con 0x7fb74409db40 2023-09-29T23:06:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74975a700 1 --2- 172.21.15.121:0/2408997027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409db40 0x7fb744147d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009e60 tx=0x7fb73c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=267c427f35f2e07c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb7437fe700 1 -- 172.21.15.121:0/2408997027 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00e070 con 0x7fb74409db40 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb7437fe700 1 -- 172.21.15.121:0/2408997027 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00bab0 con 0x7fb74409db40 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb7437fe700 1 -- 172.21.15.121:0/2408997027 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00be70 con 0x7fb74409db40 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 -- 172.21.15.121:0/2408997027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409db40 msgr2=0x7fb744147d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/2408997027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409db40 0x7fb744147d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009e60 tx=0x7fb73c009780 comp rx=0 tx=0).stop 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 -- 172.21.15.121:0/2408997027 shutdown_connections 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/2408997027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409db40 0x7fb744147d80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/2408997027 >> v2:172.21.15.123:3300/0 conn(0x7fb74409f290 0x7fb7441409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 -- 172.21.15.121:0/2408997027 >> 172.21.15.121:0/2408997027 conn(0x7fb7440a5bc0 msgr2=0x7fb7440b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 -- 172.21.15.121:0/2408997027 shutdown_connections 2023-09-29T23:06:17.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.010+0000 7fb74b1bd700 1 -- 172.21.15.121:0/2408997027 wait complete. 2023-09-29T23:06:17.014 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.011+0000 7fb74b1bd700 1 Processor -- start 2023-09-29T23:06:17.014 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74b1bd700 1 -- start start 2023-09-29T23:06:17.016 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74b1bd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744063330 0x7fb7440a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74b1bd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb74409db40 0x7fb744097390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74b1bd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409f290 0x7fb7441509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74b1bd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74409cb10 con 0x7fb74409db40 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74b1bd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744150480 con 0x7fb744063330 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74b1bd700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74406ef50 con 0x7fb74409f290 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb743fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb74409db40 0x7fb744097390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74975a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409f290 0x7fb7441509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb743fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb74409db40 0x7fb744097390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38778/0 (socket says 172.21.15.121:38778) 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb743fff700 1 -- 172.21.15.121:0/996378530 learned_addr learned my addr 172.21.15.121:0/996378530 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb748f59700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744063330 0x7fb7440a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:17.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74975a700 1 -- 172.21.15.121:0/996378530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744063330 msgr2=0x7fb7440a2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74975a700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744063330 0x7fb7440a2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74975a700 1 -- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb74409db40 msgr2=0x7fb744097390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74975a700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb74409db40 0x7fb744097390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb74975a700 1 -- 172.21.15.121:0/996378530 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c009a20 con 0x7fb74409f290 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.012+0000 7fb743fff700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb74409db40 0x7fb744097390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.013+0000 7fb74975a700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409f290 0x7fb7441509e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb73c016000 tx=0x7fb73c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.013+0000 7fb740ff9700 1 -- 172.21.15.121:0/996378530 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c012e10 con 0x7fb74409f290 2023-09-29T23:06:17.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.013+0000 7fb740ff9700 1 -- 172.21.15.121:0/996378530 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c027470 con 0x7fb74409f290 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.013+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb744097c20 con 0x7fb74409f290 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.013+0000 7fb740ff9700 1 -- 172.21.15.121:0/996378530 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c027830 con 0x7fb74409f290 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.013+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7440965b0 con 0x7fb74409f290 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.014+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb744096740 con 0x7fb74409f290 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb740ff9700 1 -- 172.21.15.121:0/996378530 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb73c024070 con 0x7fb74409f290 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb740ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb740ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb740ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb740ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:17.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb740ff9700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7280987b0 0x7fb72809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:17.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb748f59700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7280987b0 0x7fb72809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:17.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.015+0000 7fb740ff9700 1 -- 172.21.15.121:0/996378530 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3848+0+0 (secure 0 0 0) 0x7fb73c0c2700 con 0x7fb74409f290 2023-09-29T23:06:17.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.016+0000 7fb748f59700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7280987b0 0x7fb72809ac60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb738001360 tx=0x7fb738009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:17.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.019+0000 7fb740ff9700 1 -- 172.21.15.121:0/996378530 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb73c090020 con 0x7fb74409f290 2023-09-29T23:06:17.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.200+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb74406abd0 con 0x7fb74409f290 2023-09-29T23:06:17.874 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.874+0000 7fb740ff9700 1 -- 172.21.15.121:0/996378530 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v17) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb73c093640 con 0x7fb74409f290 2023-09-29T23:06:17.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.878+0000 7fb74b1bd700 10 mgrc shutdown 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7280987b0 msgr2=0x7fb72809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7280987b0 0x7fb72809ac60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb738001360 tx=0x7fb738009040 comp rx=0 tx=0).stop 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409f290 msgr2=0x7fb7441509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409f290 0x7fb7441509e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb73c016000 tx=0x7fb73c019000 comp rx=0 tx=0).stop 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 shutdown_connections 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7280987b0 0x7fb72809ac60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb74409f290 0x7fb7441509e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744063330 0x7fb7440a2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 --2- 172.21.15.121:0/996378530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb74409db40 0x7fb744097390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:17.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 >> 172.21.15.121:0/996378530 conn(0x7fb7440a5bc0 msgr2=0x7fb7441407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:17.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 shutdown_connections 2023-09-29T23:06:17.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:17.879+0000 7fb74b1bd700 1 -- 172.21.15.121:0/996378530 wait complete. 2023-09-29T23:06:17.880 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_data' created 2023-09-29T23:06:17.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-09-29T23:06:18.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.020+0000 7ff2a9763700 1 Processor -- start 2023-09-29T23:06:18.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.020+0000 7ff2a9763700 1 -- start start 2023-09-29T23:06:18.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a9763700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff2a40a5550 0x7ff2a414c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a9763700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a409a810 con 0x7ff2a40632a0 2023-09-29T23:06:18.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a9763700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a40a4ab0 con 0x7ff2a40a4570 2023-09-29T23:06:18.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a9763700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a40a4d50 con 0x7ff2a40a5550 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a2ffd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff2a40632a0 0x7ff2a409b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53892/0 (socket says 172.21.15.121:53892) 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/3654258729 learned_addr learned my addr 172.21.15.121:0/3654258729 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a37fe700 1 --2- 172.21.15.121:0/3654258729 >> v2:172.21.15.123:3300/0 conn(0x7ff2a40a5550 0x7ff2a414c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083301024 0 0) 0x7ff2a409a810 con 0x7ff2a40632a0 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.021+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff28c003570 con 0x7ff2a40632a0 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3947805172 0 0) 0x7ff28c003570 con 0x7ff2a40632a0 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 >> v2:172.21.15.123:3300/0 conn(0x7ff2a40a5550 msgr2=0x7ff2a414c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 --2- 172.21.15.121:0/3654258729 >> v2:172.21.15.123:3300/0 conn(0x7ff2a40a5550 0x7ff2a414c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 >> v1:172.21.15.123:6789/0 conn(0x7ff2a40a4570 legacy=0x7ff2a40b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a409af80 con 0x7ff2a40632a0 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7ff298001470 con 0x7ff2a40632a0 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 >> v1:172.21.15.121:6789/0 conn(0x7ff2a40632a0 legacy=0x7ff2a409b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c003a60 0x7ff28c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff28c004810 0x7ff28c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0051c0 0x7ff28c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40a4d50 con 0x7ff28c0051c0 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c00c0b0 con 0x7ff28c003a60 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c00c270 con 0x7ff28c004810 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0051c0 0x7ff28c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a27fc700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff28c004810 0x7ff28c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a2ffd700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c003a60 0x7ff28c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 -- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff28c004810 msgr2=0x7ff28c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff28c004810 0x7ff28c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 -- 172.21.15.121:0/3654258729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c003a60 msgr2=0x7ff28c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c003a60 0x7ff28c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 -- 172.21.15.121:0/3654258729 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c003570 con 0x7ff28c0051c0 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a27fc700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff28c004810 0x7ff28c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 -- 172.21.15.121:0/3654258729 reap_dead start 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a37fe700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0051c0 0x7ff28c00ba60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00e210 tx=0x7ff29c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9320eb6a603b7a6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c00eac0 con 0x7ff28c0051c0 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c00f040 con 0x7ff28c0051c0 2023-09-29T23:06:18.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.022+0000 7ff2a1ffb700 1 -- 172.21.15.121:0/3654258729 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c00ee70 con 0x7ff28c0051c0 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.023+0000 7ff2a9763700 1 -- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0051c0 msgr2=0x7ff28c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.023+0000 7ff2a9763700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0051c0 0x7ff28c00ba60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00e210 tx=0x7ff29c00b040 comp rx=0 tx=0).stop 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.023+0000 7ff2a9763700 1 -- 172.21.15.121:0/3654258729 shutdown_connections 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.023+0000 7ff2a9763700 1 --2- 172.21.15.121:0/3654258729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0051c0 0x7ff28c00ba60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.023+0000 7ff2a9763700 1 -- 172.21.15.121:0/3654258729 >> 172.21.15.121:0/3654258729 conn(0x7ff2a4069ec0 msgr2=0x7ff2a40b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.023+0000 7ff2a9763700 1 -- 172.21.15.121:0/3654258729 shutdown_connections 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.023+0000 7ff2a9763700 1 -- 172.21.15.121:0/3654258729 wait complete. 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.024+0000 7ff2a9763700 1 Processor -- start 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.024+0000 7ff2a9763700 1 -- start start 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c0051c0 0x7ff2a409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.030 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2a409d170 0x7ff2a409e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2a409e740 0x7ff2a413e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a406d930 con 0x7ff2a409e740 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40a0320 con 0x7ff28c0051c0 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a406db60 con 0x7ff2a409d170 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2a409d170 0x7ff2a409e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2a409d170 0x7ff2a409e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34864/0 (socket says 172.21.15.121:34864) 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 -- 172.21.15.121:0/92860733 learned_addr learned my addr 172.21.15.121:0/92860733 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a2ffd700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c0051c0 0x7ff2a409cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a37fe700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2a409e740 0x7ff2a413e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 -- 172.21.15.121:0/92860733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c0051c0 msgr2=0x7ff2a409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c0051c0 0x7ff2a409cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 -- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2a409e740 msgr2=0x7ff2a413e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2a409e740 0x7ff2a413e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 -- 172.21.15.121:0/92860733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a406de20 con 0x7ff2a409d170 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a27fc700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2a409d170 0x7ff2a409e200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff29400fc50 tx=0x7ff294009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff28affd700 1 -- 172.21.15.121:0/92860733 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400d920 con 0x7ff2a409d170 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff28affd700 1 -- 172.21.15.121:0/92860733 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29400da80 con 0x7ff2a409d170 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a40a2ad0 con 0x7ff2a409d170 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff28affd700 1 -- 172.21.15.121:0/92860733 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400de00 con 0x7ff2a409d170 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.025+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a413edb0 con 0x7ff2a409d170 2023-09-29T23:06:18.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.026+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a40a2930 con 0x7ff2a409d170 2023-09-29T23:06:18.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.028+0000 7ff28affd700 1 -- 172.21.15.121:0/92860733 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff294013600 con 0x7ff2a409d170 2023-09-29T23:06:18.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.028+0000 7ff28affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:18.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.028+0000 7ff28affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.028+0000 7ff28affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.028+0000 7ff28affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.028+0000 7ff28affd700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff290098780 0x7ff29009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.028+0000 7ff2a2ffd700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff290098780 0x7ff29009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.029+0000 7ff28affd700 1 -- 172.21.15.121:0/92860733 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4230+0+0 (secure 0 0 0) 0x7ff2940c1070 con 0x7ff2a409d170 2023-09-29T23:06:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.029+0000 7ff2a2ffd700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff290098780 0x7ff29009ac30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2980011e0 tx=0x7ff2a406fba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.032+0000 7ff28affd700 1 -- 172.21.15.121:0/92860733 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff29408e760 con 0x7ff2a409d170 2023-09-29T23:06:18.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.252+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff2a40a2930 con 0x7ff2a409d170 2023-09-29T23:06:18.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.257+0000 7ff28affd700 1 -- 172.21.15.121:0/92860733 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff2a40a2930 con 0x7ff2a409d170 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 10 mgrc shutdown 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff290098780 msgr2=0x7ff29009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff290098780 0x7ff29009ac30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2980011e0 tx=0x7ff2a406fba0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2a409d170 msgr2=0x7ff2a409e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2a409d170 0x7ff2a409e200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff29400fc50 tx=0x7ff294009040 comp rx=0 tx=0).stop 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 shutdown_connections 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff290098780 0x7ff29009ac30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2a409e740 0x7ff2a413e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff28c0051c0 0x7ff2a409cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 --2- 172.21.15.121:0/92860733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2a409d170 0x7ff2a409e200 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 >> 172.21.15.121:0/92860733 conn(0x7ff2a4069ec0 msgr2=0x7ff2a40b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 shutdown_connections 2023-09-29T23:06:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.260+0000 7ff2a9763700 1 -- 172.21.15.121:0/92860733 wait complete. 2023-09-29T23:06:18.261 INFO:teuthology.orchestra.run.smithi121.stderr:new fs with metadata pool 3 and data pool 4 2023-09-29T23:06:18.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:18.394 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.394+0000 7f9f748e9700 1 Processor -- start 2023-09-29T23:06:18.394 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.394+0000 7f9f748e9700 1 -- start start 2023-09-29T23:06:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f748e9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9f700b3830 0x7f9f7009e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f748e9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700b3c00 0x7f9f700a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f748e9700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7009dca0 con 0x7f9f700a3c70 2023-09-29T23:06:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f748e9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7009eed0 con 0x7f9f700b3830 2023-09-29T23:06:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f748e9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700a3740 con 0x7f9f700b3c00 2023-09-29T23:06:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6ed9d700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9f700a3c70 0x7f9f701401b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53908/0 (socket says 172.21.15.121:53908) 2023-09-29T23:06:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6dd9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700b3c00 0x7f9f700a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6ed9d700 1 -- 172.21.15.121:0/1594288050 learned_addr learned my addr 172.21.15.121:0/1594288050 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6d59a700 1 -- 172.21.15.121:0/1594288050 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285114554 0 0) 0x7f9f7009dca0 con 0x7f9f700a3c70 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6d59a700 1 -- 172.21.15.121:0/1594288050 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f58003570 con 0x7f9f700a3c70 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6e59c700 1 --2- 172.21.15.121:0/1594288050 >> v2:172.21.15.123:3300/0 conn(0x7f9f700b3830 0x7f9f7009e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6dd9b700 1 -- 172.21.15.121:0/1594288050 >> v2:172.21.15.123:3300/0 conn(0x7f9f700b3830 msgr2=0x7f9f7009e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6dd9b700 1 --2- 172.21.15.121:0/1594288050 >> v2:172.21.15.123:3300/0 conn(0x7f9f700b3830 0x7f9f7009e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6dd9b700 1 -- 172.21.15.121:0/1594288050 >> v1:172.21.15.121:6789/0 conn(0x7f9f700a3c70 legacy=0x7f9f701401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6dd9b700 1 -- 172.21.15.121:0/1594288050 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700a57c0 con 0x7f9f700b3c00 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6e59c700 1 --2- 172.21.15.121:0/1594288050 >> v2:172.21.15.123:3300/0 conn(0x7f9f700b3830 0x7f9f7009e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.395+0000 7f9f6dd9b700 1 --2- 172.21.15.121:0/1594288050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700b3c00 0x7f9f700a4f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098e0 tx=0x7f9f600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7ec898b94c4de43 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f6d59a700 1 -- 172.21.15.121:0/1594288050 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000b640 con 0x7f9f700b3c00 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f6d59a700 1 -- 172.21.15.121:0/1594288050 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6000b7a0 con 0x7f9f700b3c00 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f6d59a700 1 -- 172.21.15.121:0/1594288050 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000bb40 con 0x7f9f700b3c00 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 -- 172.21.15.121:0/1594288050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700b3c00 msgr2=0x7f9f700a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 --2- 172.21.15.121:0/1594288050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700b3c00 0x7f9f700a4f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098e0 tx=0x7f9f600091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 -- 172.21.15.121:0/1594288050 shutdown_connections 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 --2- 172.21.15.121:0/1594288050 >> v2:172.21.15.123:3300/0 conn(0x7f9f700b3830 0x7f9f7009e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 --2- 172.21.15.121:0/1594288050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700b3c00 0x7f9f700a4f40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 -- 172.21.15.121:0/1594288050 >> 172.21.15.121:0/1594288050 conn(0x7f9f700a5bc0 msgr2=0x7f9f700b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 -- 172.21.15.121:0/1594288050 shutdown_connections 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.396+0000 7f9f748e9700 1 -- 172.21.15.121:0/1594288050 wait complete. 2023-09-29T23:06:18.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.397+0000 7f9f748e9700 1 Processor -- start 2023-09-29T23:06:18.400 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.397+0000 7f9f748e9700 1 -- start start 2023-09-29T23:06:18.402 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f748e9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700a3c70 0x7f9f700698e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.402 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f748e9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f700b3830 0x7f9f7014f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.402 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f748e9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700b3c00 0x7f9f7006e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.402 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f748e9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7006df10 con 0x7f9f700b3c00 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f748e9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f70137df0 con 0x7f9f700b3830 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f748e9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7014f120 con 0x7f9f700a3c70 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700a3c70 0x7f9f700698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700a3c70 0x7f9f700698e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34896/0 (socket says 172.21.15.121:34896) 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 -- 172.21.15.121:0/864800235 learned_addr learned my addr 172.21.15.121:0/864800235 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6ed9d700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700b3c00 0x7f9f7006e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6dd9b700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f700b3830 0x7f9f7014f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 -- 172.21.15.121:0/864800235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f700b3830 msgr2=0x7f9f7014f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f700b3830 0x7f9f7014f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 -- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700b3c00 msgr2=0x7f9f7006e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700b3c00 0x7f9f7006e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 -- 172.21.15.121:0/864800235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60009420 con 0x7f9f700a3c70 2023-09-29T23:06:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6dd9b700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f700b3830 0x7f9f7014f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:06:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.398+0000 7f9f6e59c700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700a3c70 0x7f9f700698e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c830 tx=0x7f9f64008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.399+0000 7f9f5e7fc700 1 -- 172.21.15.121:0/864800235 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64005a00 con 0x7f9f700a3c70 2023-09-29T23:06:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.399+0000 7f9f5e7fc700 1 -- 172.21.15.121:0/864800235 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64005b60 con 0x7f9f700a3c70 2023-09-29T23:06:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.399+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7006e1f0 con 0x7f9f700a3c70 2023-09-29T23:06:18.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.399+0000 7f9f5e7fc700 1 -- 172.21.15.121:0/864800235 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64011600 con 0x7f9f700a3c70 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.399+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7006d470 con 0x7f9f700a3c70 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.400+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7006abd0 con 0x7f9f700a3c70 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f5e7fc700 1 -- 172.21.15.121:0/864800235 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9f64011970 con 0x7f9f700a3c70 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f5e7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f5e7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f5e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f5e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f5e7fc700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9f500987b0 0x7f9f5009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f6dd9b700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9f500987b0 0x7f9f5009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f5e7fc700 1 -- 172.21.15.121:0/864800235 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f9f640c0610 con 0x7f9f700a3c70 2023-09-29T23:06:18.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.401+0000 7f9f6dd9b700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9f500987b0 0x7f9f5009ac60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9f600093f0 tx=0x7f9f60017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.406 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.404+0000 7f9f5e7fc700 1 -- 172.21.15.121:0/864800235 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f6408dd50 con 0x7f9f700a3c70 2023-09-29T23:06:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.585+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f70071210 con 0x7f9f700a3c70 2023-09-29T23:06:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.585+0000 7f9f5e7fc700 1 -- 172.21.15.121:0/864800235 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12267 (secure 0 0 0) 0x7f9f6401e020 con 0x7f9f700a3c70 2023-09-29T23:06:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:18.585 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":18,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:18.253835+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-29T23:06:16.382138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-29T23:06:17.218163+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 10 mgrc shutdown 2023-09-29T23:06:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9f500987b0 msgr2=0x7f9f5009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9f500987b0 0x7f9f5009ac60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9f600093f0 tx=0x7f9f60017000 comp rx=0 tx=0).stop 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700a3c70 msgr2=0x7f9f700698e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700a3c70 0x7f9f700698e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c830 tx=0x7f9f64008040 comp rx=0 tx=0).stop 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 shutdown_connections 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9f500987b0 0x7f9f5009ac60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f700b3c00 0x7f9f7006e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f700a3c70 0x7f9f700698e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 --2- 172.21.15.121:0/864800235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f700b3830 0x7f9f7014f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 >> 172.21.15.121:0/864800235 conn(0x7f9f700a5bc0 msgr2=0x7f9f700b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 shutdown_connections 2023-09-29T23:06:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.587+0000 7f9f748e9700 1 -- 172.21.15.121:0/864800235 wait complete. 2023-09-29T23:06:18.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:18.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.715+0000 7efe76567700 1 Processor -- start 2023-09-29T23:06:18.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.715+0000 7efe76567700 1 -- start start 2023-09-29T23:06:18.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe76567700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe70063330 0x7efe7009f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe76567700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe7009db40 0x7efe70147d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe76567700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe70063700 con 0x7efe701409e0 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe76567700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7009dfc0 con 0x7efe7009db40 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe76567700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe70140df0 con 0x7efe70063330 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe70063330 0x7efe7009f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe70063330 0x7efe7009f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34906/0 (socket says 172.21.15.121:34906) 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 -- 172.21.15.121:0/3981214092 learned_addr learned my addr 172.21.15.121:0/3981214092 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe74b04700 1 --2- 172.21.15.121:0/3981214092 >> v2:172.21.15.123:3300/0 conn(0x7efe7009db40 0x7efe70147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 -- 172.21.15.121:0/3981214092 >> v2:172.21.15.123:3300/0 conn(0x7efe7009db40 msgr2=0x7efe70147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 --2- 172.21.15.121:0/3981214092 >> v2:172.21.15.123:3300/0 conn(0x7efe7009db40 0x7efe70147d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 -- 172.21.15.121:0/3981214092 >> v1:172.21.15.123:6789/0 conn(0x7efe701409e0 legacy=0x7efe7013ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 -- 172.21.15.121:0/3981214092 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70141050 con 0x7efe70063330 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe74b04700 1 --2- 172.21.15.121:0/3981214092 >> v2:172.21.15.123:3300/0 conn(0x7efe7009db40 0x7efe70147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6ffff700 1 --2- 172.21.15.121:0/3981214092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe70063330 0x7efe7009f290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe6400bc30 tx=0x7efe6400b4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a169e963db0e1405 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.716+0000 7efe6effd700 1 -- 172.21.15.121:0/3981214092 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6400dc50 con 0x7efe70063330 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe6effd700 1 -- 172.21.15.121:0/3981214092 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6400ddb0 con 0x7efe70063330 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe6effd700 1 -- 172.21.15.121:0/3981214092 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64005c30 con 0x7efe70063330 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 -- 172.21.15.121:0/3981214092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe70063330 msgr2=0x7efe7009f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 --2- 172.21.15.121:0/3981214092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe70063330 0x7efe7009f290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe6400bc30 tx=0x7efe6400b4c0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 -- 172.21.15.121:0/3981214092 shutdown_connections 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 --2- 172.21.15.121:0/3981214092 >> v2:172.21.15.123:3300/0 conn(0x7efe7009db40 0x7efe70147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 --2- 172.21.15.121:0/3981214092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe70063330 0x7efe7009f290 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 -- 172.21.15.121:0/3981214092 >> 172.21.15.121:0/3981214092 conn(0x7efe700a5bc0 msgr2=0x7efe700b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 -- 172.21.15.121:0/3981214092 shutdown_connections 2023-09-29T23:06:18.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.717+0000 7efe76567700 1 -- 172.21.15.121:0/3981214092 wait complete. 2023-09-29T23:06:18.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.718+0000 7efe76567700 1 Processor -- start 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe76567700 1 -- start start 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe76567700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe70063330 0x7efe70096220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe76567700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe7009db40 0x7efe70096760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe76567700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe701409e0 0x7efe7014f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe76567700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe70096ca0 con 0x7efe70063330 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe76567700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7014c9e0 con 0x7efe701409e0 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe76567700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7014cca0 con 0x7efe7009db40 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe6f7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe7009db40 0x7efe70096760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.719+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe70063330 0x7efe70096220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6f7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe7009db40 0x7efe70096760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34912/0 (socket says 172.21.15.121:34912) 2023-09-29T23:06:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6f7fe700 1 -- 172.21.15.121:0/2317349218 learned_addr learned my addr 172.21.15.121:0/2317349218 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6ffff700 1 -- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe7009db40 msgr2=0x7efe70096760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6ffff700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe7009db40 0x7efe70096760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6ffff700 1 -- 172.21.15.121:0/2317349218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe701409e0 msgr2=0x7efe7014f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6ffff700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe701409e0 0x7efe7014f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6ffff700 1 -- 172.21.15.121:0/2317349218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6400b720 con 0x7efe70063330 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6f7fe700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe7009db40 0x7efe70096760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe6ffff700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe70063330 0x7efe70096220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe64017000 tx=0x7efe6401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe57fff700 1 -- 172.21.15.121:0/2317349218 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64010680 con 0x7efe70063330 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe57fff700 1 -- 172.21.15.121:0/2317349218 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6401d070 con 0x7efe70063330 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7006eee0 con 0x7efe70063330 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe57fff700 1 -- 172.21.15.121:0/2317349218 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64010a10 con 0x7efe70063330 2023-09-29T23:06:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.720+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7006e340 con 0x7efe70063330 2023-09-29T23:06:18.724 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.721+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7006abd0 con 0x7efe70063330 2023-09-29T23:06:18.724 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe57fff700 1 -- 172.21.15.121:0/2317349218 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7efe64024440 con 0x7efe70063330 2023-09-29T23:06:18.724 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe57fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:18.724 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe57fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:18.724 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe57fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe57fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe57fff700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efe5c098780 0x7efe5c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe6f7fe700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efe5c098780 0x7efe5c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.723+0000 7efe57fff700 1 -- 172.21.15.121:0/2317349218 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4330+0+0 (secure 0 0 0) 0x7efe64014030 con 0x7efe70063330 2023-09-29T23:06:18.725 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.724+0000 7efe6f7fe700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efe5c098780 0x7efe5c09ac30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe7006bcd0 tx=0x7efe60009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:18.726 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.727+0000 7efe57fff700 1 -- 172.21.15.121:0/2317349218 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe64093070 con 0x7efe70063330 2023-09-29T23:06:18.909 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.909+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe70071210 con 0x7efe70063330 2023-09-29T23:06:18.909 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.910+0000 7efe57fff700 1 -- 172.21.15.121:0/2317349218 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12267 (secure 0 0 0) 0x7efe6401f020 con 0x7efe70063330 2023-09-29T23:06:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:18.910 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":18,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:18.253835+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-29T23:06:16.382138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-29T23:06:17.218163+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:18.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 10 mgrc shutdown 2023-09-29T23:06:18.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efe5c098780 msgr2=0x7efe5c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efe5c098780 0x7efe5c09ac30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe7006bcd0 tx=0x7efe60009040 comp rx=0 tx=0).stop 2023-09-29T23:06:18.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe70063330 msgr2=0x7efe70096220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe70063330 0x7efe70096220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe64017000 tx=0x7efe6401a000 comp rx=0 tx=0).stop 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 shutdown_connections 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efe5c098780 0x7efe5c09ac30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe701409e0 0x7efe7014f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe70063330 0x7efe70096220 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 --2- 172.21.15.121:0/2317349218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe7009db40 0x7efe70096760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 >> 172.21.15.121:0/2317349218 conn(0x7efe700a5bc0 msgr2=0x7efe7013cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 shutdown_connections 2023-09-29T23:06:18.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:18.912+0000 7efe76567700 1 -- 172.21.15.121:0/2317349218 wait complete. 2023-09-29T23:06:18.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-09-29T23:06:19.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.044+0000 7efdc878a700 1 Processor -- start 2023-09-29T23:06:19.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.044+0000 7efdc878a700 1 -- start start 2023-09-29T23:06:19.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc878a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0140f40 0x7efdc006b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:19.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc878a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efdc0148e10 0x7efdc014b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:19.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc878a700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc00a5720 con 0x7efdc006f160 2023-09-29T23:06:19.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc878a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc0140540 con 0x7efdc0148e10 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc878a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc01449e0 con 0x7efdc0140f40 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc6d27700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efdc0148e10 0x7efdc014b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc6526700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0140f40 0x7efdc006b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc6526700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0140f40 0x7efdc006b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34918/0 (socket says 172.21.15.121:34918) 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc6d27700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efdc0148e10 0x7efdc014b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38826/0 (socket says 172.21.15.121:38826) 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc6526700 1 -- 172.21.15.121:0/111164632 learned_addr learned my addr 172.21.15.121:0/111164632 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.045+0000 7efdc5d25700 1 --1- 172.21.15.121:0/111164632 >> v1:172.21.15.121:6789/0 conn(0x7efdc006f160 0x7efdc0143d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53924/0 (socket says 172.21.15.121:53924) 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc6d27700 1 -- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0140f40 msgr2=0x7efdc006b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:19.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc6d27700 1 --2- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0140f40 0x7efdc006b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc6d27700 1 -- 172.21.15.121:0/111164632 >> v1:172.21.15.121:6789/0 conn(0x7efdc006f160 legacy=0x7efdc0143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc6d27700 1 -- 172.21.15.121:0/111164632 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc0140750 con 0x7efdc0148e10 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc6526700 1 --2- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0140f40 0x7efdc006b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc6d27700 1 --2- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0148e10 0x7efdc014b1f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009e60 tx=0x7efdb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b822ce1209859145 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc5524700 1 -- 172.21.15.121:0/111164632 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb800e070 con 0x7efdc0148e10 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc5524700 1 -- 172.21.15.121:0/111164632 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb800bb60 con 0x7efdc0148e10 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc5524700 1 -- 172.21.15.121:0/111164632 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8012430 con 0x7efdc0148e10 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc878a700 1 -- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0148e10 msgr2=0x7efdc014b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc878a700 1 --2- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0148e10 0x7efdc014b1f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009e60 tx=0x7efdb8009780 comp rx=0 tx=0).stop 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc878a700 1 -- 172.21.15.121:0/111164632 shutdown_connections 2023-09-29T23:06:19.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc878a700 1 --2- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc0148e10 0x7efdc014b1f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:19.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc878a700 1 --2- 172.21.15.121:0/111164632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0140f40 0x7efdc006b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:19.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc878a700 1 -- 172.21.15.121:0/111164632 >> 172.21.15.121:0/111164632 conn(0x7efdc006d070 msgr2=0x7efdc006d470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:19.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.046+0000 7efdc878a700 1 -- 172.21.15.121:0/111164632 shutdown_connections 2023-09-29T23:06:19.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.047+0000 7efdc878a700 1 -- 172.21.15.121:0/111164632 wait complete. 2023-09-29T23:06:19.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.047+0000 7efdc878a700 1 Processor -- start 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.047+0000 7efdc878a700 1 -- start start 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc878a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc006f160 0x7efdc00b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc878a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc0140f40 0x7efdc00a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc878a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0148e10 0x7efdc00a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc878a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00a1100 con 0x7efdc006f160 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc878a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00a34a0 con 0x7efdc0140f40 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc878a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00a3700 con 0x7efdc0148e10 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6d27700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0148e10 0x7efdc00a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6526700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc006f160 0x7efdc00b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:19.052 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6d27700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0148e10 0x7efdc00a2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34928/0 (socket says 172.21.15.121:34928) 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6d27700 1 -- 172.21.15.121:0/2067181382 learned_addr learned my addr 172.21.15.121:0/2067181382 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6526700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc006f160 0x7efdc00b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38834/0 (socket says 172.21.15.121:38834) 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc5d25700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc0140f40 0x7efdc00a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6526700 1 -- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0148e10 msgr2=0x7efdc00a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6526700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0148e10 0x7efdc00a2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6526700 1 -- 172.21.15.121:0/2067181382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc0140f40 msgr2=0x7efdc00a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6526700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc0140f40 0x7efdc00a2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6526700 1 -- 172.21.15.121:0/2067181382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00094f0 con 0x7efdc006f160 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.048+0000 7efdc6d27700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0148e10 0x7efdc00a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.049+0000 7efdc6526700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc006f160 0x7efdc00b28a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efdb000ec50 tx=0x7efdb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:19.053 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.049+0000 7efdb67fc700 1 -- 172.21.15.121:0/2067181382 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000c850 con 0x7efdc006f160 2023-09-29T23:06:19.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.049+0000 7efdb67fc700 1 -- 172.21.15.121:0/2067181382 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb000c9b0 con 0x7efdc006f160 2023-09-29T23:06:19.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.049+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb8009a20 con 0x7efdc006f160 2023-09-29T23:06:19.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.049+0000 7efdb67fc700 1 -- 172.21.15.121:0/2067181382 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000cd30 con 0x7efdc006f160 2023-09-29T23:06:19.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.049+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc00a4770 con 0x7efdc006f160 2023-09-29T23:06:19.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.050+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc00643c0 con 0x7efdc006f160 2023-09-29T23:06:19.057 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.051+0000 7efdb67fc700 1 -- 172.21.15.121:0/2067181382 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7efdb0013680 con 0x7efdc006f160 2023-09-29T23:06:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.051+0000 7efdb67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.052+0000 7efdb67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.052+0000 7efdb67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.052+0000 7efdb67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:19.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.052+0000 7efdb67fc700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efda40987b0 0x7efda409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:19.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.052+0000 7efdc5d25700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efda40987b0 0x7efda409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:19.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.052+0000 7efdb67fc700 1 -- 172.21.15.121:0/2067181382 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4330+0+0 (secure 0 0 0) 0x7efdb00c0dd0 con 0x7efdc006f160 2023-09-29T23:06:19.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.052+0000 7efdc5d25700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efda40987b0 0x7efda409ac60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc007d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:19.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.055+0000 7efdb67fc700 1 -- 172.21.15.121:0/2067181382 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efdb008e470 con 0x7efdc006f160 2023-09-29T23:06:19.280 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:19.280+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7efdc00a4e60 con 0x7efdc006f160 2023-09-29T23:06:19.491 INFO:tasks.ceph.mds.b.smithi123.stderr:2023-09-29T23:06:19.491+0000 7ff1584d1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-09-29T23:06:20.281 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.281+0000 7efdb67fc700 1 -- 172.21.15.121:0/2067181382 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7efdb00200b0 con 0x7efdc006f160 2023-09-29T23:06:20.284 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 10 mgrc shutdown 2023-09-29T23:06:20.284 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efda40987b0 msgr2=0x7efda409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:20.284 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efda40987b0 0x7efda409ac60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc007d70 comp rx=0 tx=0).stop 2023-09-29T23:06:20.284 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc006f160 msgr2=0x7efdc00b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc006f160 0x7efdc00b28a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efdb000ec50 tx=0x7efdb0009250 comp rx=0 tx=0).stop 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 shutdown_connections 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efda40987b0 0x7efda409ac60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc0148e10 0x7efdc00a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc006f160 0x7efdc00b28a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 --2- 172.21.15.121:0/2067181382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc0140f40 0x7efdc00a2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 >> 172.21.15.121:0/2067181382 conn(0x7efdc006d070 msgr2=0x7efdc013d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 shutdown_connections 2023-09-29T23:06:20.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.284+0000 7efdc878a700 1 -- 172.21.15.121:0/2067181382 wait complete. 2023-09-29T23:06:20.298 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-09-29T23:06:20.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-09-29T23:06:20.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.424+0000 7f27a2064700 1 Processor -- start 2023-09-29T23:06:20.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.424+0000 7f27a2064700 1 -- start start 2023-09-29T23:06:20.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f27a2064700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f279c0634b0 0x7f279c09bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:20.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f27a2064700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f279c1468d0 0x7f279c148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:20.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f27a2064700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c0a03c0 con 0x7f279c0630e0 2023-09-29T23:06:20.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f27a2064700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c09c940 con 0x7f279c0634b0 2023-09-29T23:06:20.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f27a2064700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c09fac0 con 0x7f279c1468d0 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279affd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f279c0634b0 0x7f279c09bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279b7fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f279c0630e0 0x7f279c0b38f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53934/0 (socket says 172.21.15.121:53934) 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279b7fe700 1 -- 172.21.15.121:0/4055679805 learned_addr learned my addr 172.21.15.121:0/4055679805 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279bfff700 1 --2- 172.21.15.121:0/4055679805 >> v2:172.21.15.123:3300/0 conn(0x7f279c1468d0 0x7f279c148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279affd700 1 -- 172.21.15.121:0/4055679805 >> v2:172.21.15.123:3300/0 conn(0x7f279c1468d0 msgr2=0x7f279c148cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279affd700 1 --2- 172.21.15.121:0/4055679805 >> v2:172.21.15.123:3300/0 conn(0x7f279c1468d0 0x7f279c148cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279affd700 1 -- 172.21.15.121:0/4055679805 >> v1:172.21.15.121:6789/0 conn(0x7f279c0630e0 legacy=0x7f279c0b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279affd700 1 -- 172.21.15.121:0/4055679805 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c0a5c80 con 0x7f279c0634b0 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279a7fc700 1 -- 172.21.15.121:0/4055679805 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063128556 0 0) 0x7f279c0a03c0 con 0x7f279c0630e0 2023-09-29T23:06:20.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279affd700 1 --2- 172.21.15.121:0/4055679805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0634b0 0x7f279c09bfe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f279c06b6b0 tx=0x7f278c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7312c41dddffe4d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279a7fc700 1 -- 172.21.15.121:0/4055679805 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c00b6b0 con 0x7f279c0634b0 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279a7fc700 1 -- 172.21.15.121:0/4055679805 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278c00b810 con 0x7f279c0634b0 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.425+0000 7f279a7fc700 1 -- 172.21.15.121:0/4055679805 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c00bbb0 con 0x7f279c0634b0 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 -- 172.21.15.121:0/4055679805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0634b0 msgr2=0x7f279c09bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 --2- 172.21.15.121:0/4055679805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0634b0 0x7f279c09bfe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f279c06b6b0 tx=0x7f278c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 -- 172.21.15.121:0/4055679805 shutdown_connections 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 --2- 172.21.15.121:0/4055679805 >> v2:172.21.15.123:3300/0 conn(0x7f279c1468d0 0x7f279c148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 --2- 172.21.15.121:0/4055679805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0634b0 0x7f279c09bfe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 -- 172.21.15.121:0/4055679805 >> 172.21.15.121:0/4055679805 conn(0x7f279c069ec0 msgr2=0x7f279c0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 -- 172.21.15.121:0/4055679805 shutdown_connections 2023-09-29T23:06:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.426+0000 7f27a2064700 1 -- 172.21.15.121:0/4055679805 wait complete. 2023-09-29T23:06:20.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.427+0000 7f27a2064700 1 Processor -- start 2023-09-29T23:06:20.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.427+0000 7f27a2064700 1 -- start start 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0630e0 0x7f279c13d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f279c0634b0 0x7f279c0a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f279c1468d0 0x7f279c09d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c13d730 con 0x7f279c0630e0 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c143420 con 0x7f279c1468d0 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279c099f30 con 0x7f279c0634b0 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0630e0 0x7f279c13d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279affd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f279c0634b0 0x7f279c0a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0630e0 0x7f279c13d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38838/0 (socket says 172.21.15.121:38838) 2023-09-29T23:06:20.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 -- 172.21.15.121:0/1047278983 learned_addr learned my addr 172.21.15.121:0/1047278983 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279bfff700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f279c1468d0 0x7f279c09d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 -- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f279c0634b0 msgr2=0x7f279c0a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f279c0634b0 0x7f279c0a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 -- 172.21.15.121:0/1047278983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f279c1468d0 msgr2=0x7f279c09d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f279c1468d0 0x7f279c09d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 -- 172.21.15.121:0/1047278983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c009420 con 0x7f279c0630e0 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279bfff700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f279c1468d0 0x7f279c09d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279affd700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f279c0634b0 0x7f279c0a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f279b7fe700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0630e0 0x7f279c13d1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f279000c7e0 tx=0x7f2790007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27837fe700 1 -- 172.21.15.121:0/1047278983 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790005b30 con 0x7f279c0630e0 2023-09-29T23:06:20.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27837fe700 1 -- 172.21.15.121:0/1047278983 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2790005c90 con 0x7f279c0630e0 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279c09a1f0 con 0x7f279c0630e0 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27837fe700 1 -- 172.21.15.121:0/1047278983 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2790007300 con 0x7f279c0630e0 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.428+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c0a4b80 con 0x7f279c0630e0 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.429+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279c06ab80 con 0x7f279c0630e0 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.430+0000 7f27837fe700 1 -- 172.21.15.121:0/1047278983 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f27900076b0 con 0x7f279c0630e0 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.430+0000 7f27837fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.431+0000 7f27837fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.431+0000 7f27837fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.431+0000 7f27837fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:20.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.431+0000 7f27837fe700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2788098780 0x7f278809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:20.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.431+0000 7f279affd700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2788098780 0x7f278809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:20.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.431+0000 7f27837fe700 1 -- 172.21.15.121:0/1047278983 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f27900bfd90 con 0x7f279c0630e0 2023-09-29T23:06:20.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.431+0000 7f279affd700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2788098780 0x7f278809ac30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f279c06b6b0 tx=0x7f278c017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:20.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.434+0000 7f27837fe700 1 -- 172.21.15.121:0/1047278983 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f279008d4d0 con 0x7f279c0630e0 2023-09-29T23:06:20.658 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:20.658+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f279c071660 con 0x7f279c0630e0 2023-09-29T23:06:21.282 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.283+0000 7f27837fe700 1 -- 172.21.15.121:0/1047278983 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2790017020 con 0x7f279c0630e0 2023-09-29T23:06:21.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 10 mgrc shutdown 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2788098780 msgr2=0x7f278809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2788098780 0x7f278809ac30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f279c06b6b0 tx=0x7f278c017000 comp rx=0 tx=0).stop 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0630e0 msgr2=0x7f279c13d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0630e0 0x7f279c13d1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f279000c7e0 tx=0x7f2790007df0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 shutdown_connections 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2788098780 0x7f278809ac30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f279c1468d0 0x7f279c09d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f279c0630e0 0x7f279c13d1f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.285+0000 7f27a2064700 1 --2- 172.21.15.121:0/1047278983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f279c0634b0 0x7f279c0a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.286 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.286+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 >> 172.21.15.121:0/1047278983 conn(0x7f279c069ec0 msgr2=0x7f279c09bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:21.287 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.286+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 shutdown_connections 2023-09-29T23:06:21.287 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.286+0000 7f27a2064700 1 -- 172.21.15.121:0/1047278983 wait complete. 2023-09-29T23:06:21.300 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-09-29T23:06:21.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-29T23:06:21.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.423+0000 7f61e44bb700 1 Processor -- start 2023-09-29T23:06:21.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.423+0000 7f61e44bb700 1 -- start start 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e44bb700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f61dc0a3cd0 0x7f61dc0a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e44bb700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc063700 con 0x7f61dc09f820 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e44bb700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc0a0880 con 0x7f61dc063330 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e44bb700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61dc0a57e0 con 0x7f61dc0a3cd0 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e1a56700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f61dc09f820 0x7f61dc0a2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53936/0 (socket says 172.21.15.121:53936) 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e1a56700 1 -- 172.21.15.121:0/115110635 learned_addr learned my addr 172.21.15.121:0/115110635 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e2a58700 1 --2- 172.21.15.121:0/115110635 >> v2:172.21.15.121:3300/0 conn(0x7f61dc0a3cd0 0x7f61dc0a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e1255700 1 -- 172.21.15.121:0/115110635 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554046380 0 0) 0x7f61dc063700 con 0x7f61dc09f820 2023-09-29T23:06:21.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e1255700 1 -- 172.21.15.121:0/115110635 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c8003570 con 0x7f61dc09f820 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e2a58700 1 -- 172.21.15.121:0/115110635 >> v1:172.21.15.123:6789/0 conn(0x7f61dc063330 legacy=0x7f61dc0a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e2a58700 1 -- 172.21.15.121:0/115110635 >> v1:172.21.15.121:6789/0 conn(0x7f61dc09f820 legacy=0x7f61dc0a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e2a58700 1 -- 172.21.15.121:0/115110635 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc0a4250 con 0x7f61dc0a3cd0 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e2a58700 1 --2- 172.21.15.121:0/115110635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc0a3cd0 0x7f61dc0a2a70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61d4009e60 tx=0x7f61d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764dbb97f978c388 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e1255700 1 -- 172.21.15.121:0/115110635 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d400e070 con 0x7f61dc0a3cd0 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e1255700 1 -- 172.21.15.121:0/115110635 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d400bb60 con 0x7f61dc0a3cd0 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.424+0000 7f61e1255700 1 -- 172.21.15.121:0/115110635 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4012430 con 0x7f61dc0a3cd0 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.425+0000 7f61e44bb700 1 -- 172.21.15.121:0/115110635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc0a3cd0 msgr2=0x7f61dc0a2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.425+0000 7f61e44bb700 1 --2- 172.21.15.121:0/115110635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc0a3cd0 0x7f61dc0a2a70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61d4009e60 tx=0x7f61d4009780 comp rx=0 tx=0).stop 2023-09-29T23:06:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.425+0000 7f61e44bb700 1 -- 172.21.15.121:0/115110635 shutdown_connections 2023-09-29T23:06:21.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.425+0000 7f61e44bb700 1 --2- 172.21.15.121:0/115110635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc0a3cd0 0x7f61dc0a2a70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.425+0000 7f61e44bb700 1 -- 172.21.15.121:0/115110635 >> 172.21.15.121:0/115110635 conn(0x7f61dc0a5bc0 msgr2=0x7f61dc0b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:21.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.425+0000 7f61e44bb700 1 -- 172.21.15.121:0/115110635 shutdown_connections 2023-09-29T23:06:21.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.425+0000 7f61e44bb700 1 -- 172.21.15.121:0/115110635 wait complete. 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 Processor -- start 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 -- start start 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61dc063330 0x7f61dc099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc09f820 0x7f61dc0984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61dc0a3cd0 0x7f61dc13f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc09a900 con 0x7f61dc09f820 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc09b300 con 0x7f61dc0a3cd0 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e44bb700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc098f20 con 0x7f61dc063330 2023-09-29T23:06:21.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e1a56700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc09f820 0x7f61dc0984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e2257700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61dc063330 0x7f61dc099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e1a56700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc09f820 0x7f61dc0984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38858/0 (socket says 172.21.15.121:38858) 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.426+0000 7f61e1a56700 1 -- 172.21.15.121:0/1388996658 learned_addr learned my addr 172.21.15.121:0/1388996658 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e2a58700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61dc0a3cd0 0x7f61dc13f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e1a56700 1 -- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61dc063330 msgr2=0x7f61dc099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e1a56700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61dc063330 0x7f61dc099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e1a56700 1 -- 172.21.15.121:0/1388996658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61dc0a3cd0 msgr2=0x7f61dc13f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e1a56700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61dc0a3cd0 0x7f61dc13f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e1a56700 1 -- 172.21.15.121:0/1388996658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d4009a20 con 0x7f61dc09f820 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e2257700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61dc063330 0x7f61dc099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:21.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e1a56700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc09f820 0x7f61dc0984e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00c7c0 tx=0x7f61cc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61d27fc700 1 -- 172.21.15.121:0/1388996658 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc005cd0 con 0x7f61dc09f820 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61d27fc700 1 -- 172.21.15.121:0/1388996658 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc005e30 con 0x7f61dc09f820 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc097e20 con 0x7f61dc09f820 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61d27fc700 1 -- 172.21.15.121:0/1388996658 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc007320 con 0x7f61dc09f820 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.427+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc09a5f0 con 0x7f61dc09f820 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.428+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc06abd0 con 0x7f61dc09f820 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.429+0000 7f61d27fc700 1 -- 172.21.15.121:0/1388996658 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f61cc01b3b0 con 0x7f61dc09f820 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.429+0000 7f61d27fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.430+0000 7f61d27fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.430+0000 7f61d27fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:21.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.430+0000 7f61d27fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:21.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.430+0000 7f61d27fc700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f61c0098780 0x7f61c009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.430+0000 7f61e2257700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f61c0098780 0x7f61c009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.430+0000 7f61d27fc700 1 -- 172.21.15.121:0/1388996658 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f61cc0bff20 con 0x7f61dc09f820 2023-09-29T23:06:21.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.430+0000 7f61e2257700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f61c0098780 0x7f61c009ac30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61d8007fd0 tx=0x7f61d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:21.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.433+0000 7f61d27fc700 1 -- 172.21.15.121:0/1388996658 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61cc08d5d0 con 0x7f61dc09f820 2023-09-29T23:06:21.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.656+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61dc09b4c0 con 0x7f61dc09f820 2023-09-29T23:06:21.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.656+0000 7f61d27fc700 1 -- 172.21.15.121:0/1388996658 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+3841 (secure 0 0 0) 0x7f61cc017070 con 0x7f61dc09f820 2023-09-29T23:06:21.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:21.657 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6811/3133589684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3133589684},{"type":"v1","addr":"172.21.15.121:6811","nonce":3133589684}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6813/190961274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":190961274},{"type":"v1","addr":"172.21.15.121:6813","nonce":190961274}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-09-29T23:06:18.254127+0000","modified":"2023-09-29T23:06:21.280286+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4240":{"gid":4240,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6825/4268281148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":4268281148},{"type":"v1","addr":"172.21.15.123:6825","nonce":4268281148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.658+0000 7f61e44bb700 10 mgrc shutdown 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f61c0098780 msgr2=0x7f61c009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f61c0098780 0x7f61c009ac30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61d8007fd0 tx=0x7f61d8009040 comp rx=0 tx=0).stop 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc09f820 msgr2=0x7f61dc0984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc09f820 0x7f61dc0984e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00c7c0 tx=0x7f61cc007e00 comp rx=0 tx=0).stop 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 shutdown_connections 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f61c0098780 0x7f61c009ac30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61dc0a3cd0 0x7f61dc13f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61dc063330 0x7f61dc099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 --2- 172.21.15.121:0/1388996658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61dc09f820 0x7f61dc0984e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.660 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 >> 172.21.15.121:0/1388996658 conn(0x7f61dc0a5bc0 msgr2=0x7f61dc141dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:21.660 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 shutdown_connections 2023-09-29T23:06:21.660 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.659+0000 7f61e44bb700 1 -- 172.21.15.121:0/1388996658 wait complete. 2023-09-29T23:06:21.660 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 8 2023-09-29T23:06:21.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:21.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 Processor -- start 2023-09-29T23:06:21.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 -- start start 2023-09-29T23:06:21.789 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1dd80630e0 0x7f1dd8142000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.789 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1dd8142540 0x7f1dd809ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.789 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80634b0 0x7f1dd809b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.789 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd809b880 con 0x7f1dd8142540 2023-09-29T23:06:21.789 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd81429d0 con 0x7f1dd80630e0 2023-09-29T23:06:21.789 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1ddd710700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8142c90 con 0x7f1dd80634b0 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.787+0000 7f1dd67fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1dd8142540 0x7f1dd809ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd77fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80634b0 0x7f1dd809b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd67fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1dd8142540 0x7f1dd809ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38872/0 (socket says 172.21.15.121:38872) 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd67fc700 1 -- 172.21.15.121:0/2783872175 learned_addr learned my addr 172.21.15.121:0/2783872175 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd6ffd700 1 --2- 172.21.15.121:0/2783872175 >> v2:172.21.15.123:3300/0 conn(0x7f1dd80630e0 0x7f1dd8142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd77fe700 1 -- 172.21.15.121:0/2783872175 >> v2:172.21.15.123:3300/0 conn(0x7f1dd80630e0 msgr2=0x7f1dd8142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd77fe700 1 --2- 172.21.15.121:0/2783872175 >> v2:172.21.15.123:3300/0 conn(0x7f1dd80630e0 0x7f1dd8142000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd77fe700 1 -- 172.21.15.121:0/2783872175 >> v2:172.21.15.121:3300/0 conn(0x7f1dd8142540 msgr2=0x7f1dd809ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd77fe700 1 --2- 172.21.15.121:0/2783872175 >> v2:172.21.15.121:3300/0 conn(0x7f1dd8142540 0x7f1dd809ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd77fe700 1 -- 172.21.15.121:0/2783872175 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8136850 con 0x7f1dd80634b0 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd67fc700 1 --2- 172.21.15.121:0/2783872175 >> v2:172.21.15.121:3300/0 conn(0x7f1dd8142540 0x7f1dd809ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd77fe700 1 --2- 172.21.15.121:0/2783872175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80634b0 0x7f1dd809b340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80b3b50 tx=0x7f1dd0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a37949d851db954f server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd5ffb700 1 -- 172.21.15.121:0/2783872175 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000e070 con 0x7f1dd80634b0 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd5ffb700 1 -- 172.21.15.121:0/2783872175 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd000bab0 con 0x7f1dd80634b0 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.788+0000 7f1dd5ffb700 1 -- 172.21.15.121:0/2783872175 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000be70 con 0x7f1dd80634b0 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 -- 172.21.15.121:0/2783872175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80634b0 msgr2=0x7f1dd809b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2783872175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80634b0 0x7f1dd809b340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80b3b50 tx=0x7f1dd0009780 comp rx=0 tx=0).stop 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 -- 172.21.15.121:0/2783872175 shutdown_connections 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2783872175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80634b0 0x7f1dd809b340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2783872175 >> v2:172.21.15.123:3300/0 conn(0x7f1dd80630e0 0x7f1dd8142000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2783872175 >> v2:172.21.15.121:3300/0 conn(0x7f1dd8142540 0x7f1dd809ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 -- 172.21.15.121:0/2783872175 >> 172.21.15.121:0/2783872175 conn(0x7f1dd80a5bc0 msgr2=0x7f1dd80b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:21.792 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 -- 172.21.15.121:0/2783872175 shutdown_connections 2023-09-29T23:06:21.792 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.789+0000 7f1ddd710700 1 -- 172.21.15.121:0/2783872175 wait complete. 2023-09-29T23:06:21.794 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.790+0000 7f1ddd710700 1 Processor -- start 2023-09-29T23:06:21.794 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.790+0000 7f1ddd710700 1 -- start start 2023-09-29T23:06:21.794 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1ddd710700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80630e0 0x7f1dd81361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.794 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1ddd710700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dd80634b0 0x7f1dd806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1ddd710700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dd8142540 0x7f1dd806d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1ddd710700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8135d50 con 0x7f1dd8142540 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1ddd710700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd809e250 con 0x7f1dd80634b0 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1ddd710700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd809e510 con 0x7f1dd80630e0 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80630e0 0x7f1dd81361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80630e0 0x7f1dd81361e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34964/0 (socket says 172.21.15.121:34964) 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 -- 172.21.15.121:0/2592079326 learned_addr learned my addr 172.21.15.121:0/2592079326 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd77fe700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dd8142540 0x7f1dd806d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd67fc700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dd80634b0 0x7f1dd806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 -- 172.21.15.121:0/2592079326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dd80634b0 msgr2=0x7f1dd806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dd80634b0 0x7f1dd806cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 -- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dd8142540 msgr2=0x7f1dd806d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dd8142540 0x7f1dd806d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 -- 172.21.15.121:0/2592079326 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc0094f0 con 0x7f1dd80630e0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd77fe700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dd8142540 0x7f1dd806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.791+0000 7f1dd6ffd700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80630e0 0x7f1dd81361e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800ea00 tx=0x7f1dc8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.792+0000 7f1dbeffd700 1 -- 172.21.15.121:0/2592079326 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc800c850 con 0x7f1dd80630e0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.792+0000 7f1dbeffd700 1 -- 172.21.15.121:0/2592079326 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc800c9b0 con 0x7f1dd80630e0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.792+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd0009a20 con 0x7f1dd80630e0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.792+0000 7f1dbeffd700 1 -- 172.21.15.121:0/2592079326 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc800cd30 con 0x7f1dd80630e0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.792+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd809dea0 con 0x7f1dd80630e0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.792+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd80b13c0 con 0x7f1dd80630e0 2023-09-29T23:06:21.796 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.793+0000 7f1dbeffd700 1 -- 172.21.15.121:0/2592079326 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f1dc8013680 con 0x7f1dd80630e0 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.793+0000 7f1dbeffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.794+0000 7f1dbeffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.794+0000 7f1dbeffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.794+0000 7f1dbeffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.794+0000 7f1dbeffd700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1dc4098780 0x7f1dc409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.794+0000 7f1dd67fc700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1dc4098780 0x7f1dc409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.794+0000 7f1dbeffd700 1 -- 172.21.15.121:0/2592079326 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f1dc80c09d0 con 0x7f1dd80630e0 2023-09-29T23:06:21.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.794+0000 7f1dd67fc700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1dc4098780 0x7f1dc409ac30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80a2f00 tx=0x7f1dcc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:21.798 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.798+0000 7f1dbeffd700 1 -- 172.21.15.121:0/2592079326 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1dc808e110 con 0x7f1dd80630e0 2023-09-29T23:06:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.977+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dd8136030 con 0x7f1dd80630e0 2023-09-29T23:06:21.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.977+0000 7f1dbeffd700 1 -- 172.21.15.121:0/2592079326 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12254 (secure 0 0 0) 0x7f1dc8017090 con 0x7f1dd80630e0 2023-09-29T23:06:21.977 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:21.977 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":19,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:19.255324+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-29T23:06:16.382138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-29T23:06:17.218163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 10 mgrc shutdown 2023-09-29T23:06:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1dc4098780 msgr2=0x7f1dc409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1dc4098780 0x7f1dc409ac30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80a2f00 tx=0x7f1dcc008040 comp rx=0 tx=0).stop 2023-09-29T23:06:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80630e0 msgr2=0x7f1dd81361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80630e0 0x7f1dd81361e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800ea00 tx=0x7f1dc8009120 comp rx=0 tx=0).stop 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 shutdown_connections 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1dc4098780 0x7f1dc409ac30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dd8142540 0x7f1dd806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1dd80630e0 0x7f1dd81361e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 --2- 172.21.15.121:0/2592079326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dd80634b0 0x7f1dd806cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 >> 172.21.15.121:0/2592079326 conn(0x7f1dd80a5bc0 msgr2=0x7f1dd809abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 shutdown_connections 2023-09-29T23:06:21.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:21.979+0000 7f1ddd710700 1 -- 172.21.15.121:0/2592079326 wait complete. 2023-09-29T23:06:21.991 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-09-29T23:06:21.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:06:22.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.109+0000 7f7d457a1700 1 Processor -- start 2023-09-29T23:06:22.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.109+0000 7f7d457a1700 1 -- start start 2023-09-29T23:06:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d457a1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4006ccc0 0x7f7d40134140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d457a1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d40134680 0x7f7d401419e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d457a1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4009ee30 con 0x7f7d40067690 2023-09-29T23:06:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d457a1700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d40067a60 con 0x7f7d40134680 2023-09-29T23:06:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d457a1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40134ad0 con 0x7f7d4006ccc0 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d3f7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4006ccc0 0x7f7d40134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d40134680 0x7f7d401419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d40134680 0x7f7d401419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38888/0 (socket says 172.21.15.121:38888) 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d3f7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4006ccc0 0x7f7d40134140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34980/0 (socket says 172.21.15.121:34980) 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 -- 172.21.15.121:0/2242524741 learned_addr learned my addr 172.21.15.121:0/2242524741 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 -- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4006ccc0 msgr2=0x7f7d40134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 --2- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4006ccc0 0x7f7d40134140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 -- 172.21.15.121:0/2242524741 >> v1:172.21.15.123:6789/0 conn(0x7f7d40067690 legacy=0x7f7d4009fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 -- 172.21.15.121:0/2242524741 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40142670 con 0x7f7d40134680 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d3f7fe700 1 --2- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4006ccc0 0x7f7d40134140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d44fa0700 1 --2- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d40134680 0x7f7d401419e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009f20 tx=0x7f7d38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cfda8bacede724 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d3effd700 1 -- 172.21.15.121:0/2242524741 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3800e070 con 0x7f7d40134680 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d3effd700 1 -- 172.21.15.121:0/2242524741 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3800b9d0 con 0x7f7d40134680 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.110+0000 7f7d3effd700 1 -- 172.21.15.121:0/2242524741 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d380092e0 con 0x7f7d40134680 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 -- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d40134680 msgr2=0x7f7d401419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 --2- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d40134680 0x7f7d401419e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009f20 tx=0x7f7d38009840 comp rx=0 tx=0).stop 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 -- 172.21.15.121:0/2242524741 shutdown_connections 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 --2- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d40134680 0x7f7d401419e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 --2- 172.21.15.121:0/2242524741 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4006ccc0 0x7f7d40134140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 -- 172.21.15.121:0/2242524741 >> 172.21.15.121:0/2242524741 conn(0x7f7d400b3770 msgr2=0x7f7d400b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 -- 172.21.15.121:0/2242524741 shutdown_connections 2023-09-29T23:06:22.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.111+0000 7f7d457a1700 1 -- 172.21.15.121:0/2242524741 wait complete. 2023-09-29T23:06:22.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.112+0000 7f7d457a1700 1 Processor -- start 2023-09-29T23:06:22.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.112+0000 7f7d457a1700 1 -- start start 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d40067690 0x7f7d40145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4006ccc0 0x7f7d4009b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d40134680 0x7f7d400b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400a3fa0 con 0x7f7d4006ccc0 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40146450 con 0x7f7d40134680 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d401466f0 con 0x7f7d40067690 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d40067690 0x7f7d40145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4006ccc0 0x7f7d4009b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4006ccc0 0x7f7d4009b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38904/0 (socket says 172.21.15.121:38904) 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3ffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d40067690 0x7f7d40145480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34986/0 (socket says 172.21.15.121:34986) 2023-09-29T23:06:22.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 -- 172.21.15.121:0/1879400228 learned_addr learned my addr 172.21.15.121:0/1879400228 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d44fa0700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d40134680 0x7f7d400b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 -- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d40067690 msgr2=0x7f7d40145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d40067690 0x7f7d40145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 -- 172.21.15.121:0/1879400228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d40134680 msgr2=0x7f7d400b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d40134680 0x7f7d400b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 -- 172.21.15.121:0/1879400228 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d30009420 con 0x7f7d4006ccc0 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d3f7fe700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4006ccc0 0x7f7d4009b280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000ec80 tx=0x7f7d300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d27fff700 1 -- 172.21.15.121:0/1879400228 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000c850 con 0x7f7d4006ccc0 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d27fff700 1 -- 172.21.15.121:0/1879400228 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3000c9b0 con 0x7f7d4006ccc0 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d38009ae0 con 0x7f7d4006ccc0 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d27fff700 1 -- 172.21.15.121:0/1879400228 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000cd50 con 0x7f7d4006ccc0 2023-09-29T23:06:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.113+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d40145b30 con 0x7f7d4006ccc0 2023-09-29T23:06:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.114+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4006d9b0 con 0x7f7d4006ccc0 2023-09-29T23:06:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d27fff700 1 -- 172.21.15.121:0/1879400228 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7d30013680 con 0x7f7d4006ccc0 2023-09-29T23:06:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d27fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d27fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d27fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d27fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d27fff700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7d2c098780 0x7f7d2c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d3ffff700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7d2c098780 0x7f7d2c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.116+0000 7f7d27fff700 1 -- 172.21.15.121:0/1879400228 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f7d300c0ff0 con 0x7f7d4006ccc0 2023-09-29T23:06:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.117+0000 7f7d3ffff700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7d2c098780 0x7f7d2c09ac30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d34003a80 tx=0x7f7d34007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.120+0000 7f7d27fff700 1 -- 172.21.15.121:0/1879400228 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d3008e6b0 con 0x7f7d4006ccc0 2023-09-29T23:06:22.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.297+0000 7f7d457a1700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-29T23:06:22.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.297+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 --> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d4009af70 con 0x7f7d2c098780 2023-09-29T23:06:22.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.303+0000 7f7d27fff700 1 -- 172.21.15.121:0/1879400228 <== mgr.4107 v2:172.21.15.121:6808/113227 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259742 (secure 0 0 0) 0x7f7d4009af70 con 0x7f7d2c098780 2023-09-29T23:06:22.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.303+0000 7f7d27fff700 20 mgrc handle_command_reply tid 0 r 0 2023-09-29T23:06:22.306 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 10 mgrc shutdown 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7d2c098780 msgr2=0x7f7d2c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7d2c098780 0x7f7d2c09ac30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d34003a80 tx=0x7f7d34007ec0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4006ccc0 msgr2=0x7f7d4009b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4006ccc0 0x7f7d4009b280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000ec80 tx=0x7f7d300091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 shutdown_connections 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7d2c098780 0x7f7d2c09ac30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d40134680 0x7f7d400b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d40067690 0x7f7d40145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 --2- 172.21.15.121:0/1879400228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4006ccc0 0x7f7d4009b280 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 >> 172.21.15.121:0/1879400228 conn(0x7f7d400b3770 msgr2=0x7f7d40070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 shutdown_connections 2023-09-29T23:06:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.307+0000 7f7d457a1700 1 -- 172.21.15.121:0/1879400228 wait complete. 2023-09-29T23:06:22.310 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2023-09-29T23:06:22.318 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-09-29T23:06:21.486549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":33744,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":33088,"kb_avail":377453616,"statfs":{"total":386547056640,"available":386512502784,"internally_reserved":0,"allocated":638976,"data_stored":1344008,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":33882112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.622899"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869376+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868073+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.872445+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866503+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869137+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866518+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869342+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871195+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866410+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868480+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868259+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868869+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868303+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869355+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868341+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.872531+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868258+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871245+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871420+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:11.325849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383404+0000","last_change":"2023-09-29T23:06:13.639723+0000","last_active":"2023-09-29T23:06:14.383404+0000","last_peered":"2023-09-29T23:06:14.383404+0000","last_clean":"2023-09-29T23:06:14.383404+0000","last_became_active":"2023-09-29T23:06:13.639406+0000","last_became_peered":"2023-09-29T23:06:13.639406+0000","last_unstale":"2023-09-29T23:06:14.383404+0000","last_undegraded":"2023-09-29T23:06:14.383404+0000","last_fullsized":"2023-09-29T23:06:14.383404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:07:23.007465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868356+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380735+0000","last_change":"2023-09-29T23:06:13.380282+0000","last_active":"2023-09-29T23:06:14.380735+0000","last_peered":"2023-09-29T23:06:14.380735+0000","last_clean":"2023-09-29T23:06:14.380735+0000","last_became_active":"2023-09-29T23:06:13.379948+0000","last_became_peered":"2023-09-29T23:06:13.379948+0000","last_unstale":"2023-09-29T23:06:14.380735+0000","last_undegraded":"2023-09-29T23:06:14.380735+0000","last_fullsized":"2023-09-29T23:06:14.380735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:02.759420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869077+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.930970+0000","last_change":"2023-09-29T23:06:13.639753+0000","last_active":"2023-09-29T23:06:14.930970+0000","last_peered":"2023-09-29T23:06:14.930970+0000","last_clean":"2023-09-29T23:06:14.930970+0000","last_became_active":"2023-09-29T23:06:13.639436+0000","last_became_peered":"2023-09-29T23:06:13.639436+0000","last_unstale":"2023-09-29T23:06:14.930970+0000","last_undegraded":"2023-09-29T23:06:14.930970+0000","last_fullsized":"2023-09-29T23:06:14.930970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:09:57.060849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866385+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380753+0000","last_change":"2023-09-29T23:06:13.380967+0000","last_active":"2023-09-29T23:06:14.380753+0000","last_peered":"2023-09-29T23:06:14.380753+0000","last_clean":"2023-09-29T23:06:14.380753+0000","last_became_active":"2023-09-29T23:06:13.379890+0000","last_became_peered":"2023-09-29T23:06:13.379890+0000","last_unstale":"2023-09-29T23:06:14.380753+0000","last_undegraded":"2023-09-29T23:06:14.380753+0000","last_fullsized":"2023-09-29T23:06:14.380753+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:46.738824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'117","reported_seq":163,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383325+0000","last_change":"2023-09-29T23:06:09.717533+0000","last_active":"2023-09-29T23:06:14.383325+0000","last_peered":"2023-09-29T23:06:14.383325+0000","last_clean":"2023-09-29T23:06:14.383325+0000","last_became_active":"2023-09-29T23:06:09.716536+0000","last_became_peered":"2023-09-29T23:06:09.716536+0000","last_unstale":"2023-09-29T23:06:14.383325+0000","last_undegraded":"2023-09-29T23:06:14.383325+0000","last_fullsized":"2023-09-29T23:06:14.383325+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_clean_scrub_stamp":"2023-09-29T23:06:08.290533+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:44:31.128783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871160+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380737+0000","last_change":"2023-09-29T23:06:13.380209+0000","last_active":"2023-09-29T23:06:14.380737+0000","last_peered":"2023-09-29T23:06:14.380737+0000","last_clean":"2023-09-29T23:06:14.380737+0000","last_became_active":"2023-09-29T23:06:13.379845+0000","last_became_peered":"2023-09-29T23:06:13.379845+0000","last_unstale":"2023-09-29T23:06:14.380737+0000","last_undegraded":"2023-09-29T23:06:14.380737+0000","last_fullsized":"2023-09-29T23:06:14.380737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866580+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380734+0000","last_change":"2023-09-29T23:06:13.380613+0000","last_active":"2023-09-29T23:06:14.380734+0000","last_peered":"2023-09-29T23:06:14.380734+0000","last_clean":"2023-09-29T23:06:14.380734+0000","last_became_active":"2023-09-29T23:06:13.380347+0000","last_became_peered":"2023-09-29T23:06:13.380347+0000","last_unstale":"2023-09-29T23:06:14.380734+0000","last_undegraded":"2023-09-29T23:06:14.380734+0000","last_fullsized":"2023-09-29T23:06:14.380734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:35.568211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866389+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871592+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383367+0000","last_change":"2023-09-29T23:06:13.379718+0000","last_active":"2023-09-29T23:06:14.383367+0000","last_peered":"2023-09-29T23:06:14.383367+0000","last_clean":"2023-09-29T23:06:14.383367+0000","last_became_active":"2023-09-29T23:06:13.379450+0000","last_became_peered":"2023-09-29T23:06:13.379450+0000","last_unstale":"2023-09-29T23:06:14.383367+0000","last_undegraded":"2023-09-29T23:06:14.383367+0000","last_fullsized":"2023-09-29T23:06:14.383367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380884+0000","last_change":"2023-09-29T23:06:13.382829+0000","last_active":"2023-09-29T23:06:14.380884+0000","last_peered":"2023-09-29T23:06:14.380884+0000","last_clean":"2023-09-29T23:06:14.380884+0000","last_became_active":"2023-09-29T23:06:13.382590+0000","last_became_peered":"2023-09-29T23:06:13.382590+0000","last_unstale":"2023-09-29T23:06:14.380884+0000","last_undegraded":"2023-09-29T23:06:14.380884+0000","last_fullsized":"2023-09-29T23:06:14.380884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:06.822425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866864+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.873104+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868072+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.870113+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:29:04.705048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871685+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868072+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8136,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363704,"statfs":{"total":96636764160,"available":96628432896,"internally_reserved":0,"allocated":131072,"data_stored":40818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8704,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94363136,"statfs":{"total":96636764160,"available":96627851264,"internally_reserved":0,"allocated":188416,"data_stored":631186,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":131072,"data_stored":40818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8704,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94363136,"statfs":{"total":96636764160,"available":96627851264,"internally_reserved":0,"allocated":188416,"data_stored":631186,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:06:22.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:06:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.475+0000 7f35a6a05700 1 Processor -- start 2023-09-29T23:06:22.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.475+0000 7f35a6a05700 1 -- start start 2023-09-29T23:06:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f35a6a05700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a00630e0 0x7f35a0142000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f35a6a05700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f35a00634b0 0x7f35a009b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f35a6a05700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a01429d0 con 0x7f35a0142540 2023-09-29T23:06:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f35a6a05700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a0142c90 con 0x7f35a00634b0 2023-09-29T23:06:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f35a6a05700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0143f60 con 0x7f35a00630e0 2023-09-29T23:06:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359ffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a00630e0 0x7f35a0142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359f7fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f35a0142540 0x7f35a009ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53948/0 (socket says 172.21.15.121:53948) 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359f7fe700 1 -- 172.21.15.121:0/3937880710 learned_addr learned my addr 172.21.15.121:0/3937880710 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f35a4fa2700 1 --2- 172.21.15.121:0/3937880710 >> v2:172.21.15.123:3300/0 conn(0x7f35a00634b0 0x7f35a009b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359effd700 1 -- 172.21.15.121:0/3937880710 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065804987 0 0) 0x7f35a01429d0 con 0x7f35a0142540 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359effd700 1 -- 172.21.15.121:0/3937880710 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3588003570 con 0x7f35a0142540 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359ffff700 1 -- 172.21.15.121:0/3937880710 >> v2:172.21.15.123:3300/0 conn(0x7f35a00634b0 msgr2=0x7f35a009b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359ffff700 1 --2- 172.21.15.121:0/3937880710 >> v2:172.21.15.123:3300/0 conn(0x7f35a00634b0 0x7f35a009b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359ffff700 1 -- 172.21.15.121:0/3937880710 >> v1:172.21.15.121:6789/0 conn(0x7f35a0142540 legacy=0x7f35a009ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359ffff700 1 -- 172.21.15.121:0/3937880710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a01440a0 con 0x7f35a00630e0 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359ffff700 1 --2- 172.21.15.121:0/3937880710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a00630e0 0x7f35a0142000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f35940098d0 tx=0x7f3594009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55e2e48feeb72eb server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.478 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359effd700 1 -- 172.21.15.121:0/3937880710 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594005c10 con 0x7f35a00630e0 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359effd700 1 -- 172.21.15.121:0/3937880710 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594005d70 con 0x7f35a00630e0 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.476+0000 7f359effd700 1 -- 172.21.15.121:0/3937880710 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594006240 con 0x7f35a00630e0 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 -- 172.21.15.121:0/3937880710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a00630e0 msgr2=0x7f35a0142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 --2- 172.21.15.121:0/3937880710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a00630e0 0x7f35a0142000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f35940098d0 tx=0x7f3594009120 comp rx=0 tx=0).stop 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 -- 172.21.15.121:0/3937880710 shutdown_connections 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 --2- 172.21.15.121:0/3937880710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a00630e0 0x7f35a0142000 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 --2- 172.21.15.121:0/3937880710 >> v2:172.21.15.123:3300/0 conn(0x7f35a00634b0 0x7f35a009b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 -- 172.21.15.121:0/3937880710 >> 172.21.15.121:0/3937880710 conn(0x7f35a00a5bc0 msgr2=0x7f35a00b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 -- 172.21.15.121:0/3937880710 shutdown_connections 2023-09-29T23:06:22.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.477+0000 7f35a6a05700 1 -- 172.21.15.121:0/3937880710 wait complete. 2023-09-29T23:06:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.478+0000 7f35a6a05700 1 Processor -- start 2023-09-29T23:06:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.478+0000 7f35a6a05700 1 -- start start 2023-09-29T23:06:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a00630e0 0x7f35a0134190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35a00634b0 0x7f35a0135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a0142540 0x7f35a00a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a009f690 con 0x7f35a00630e0 2023-09-29T23:06:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0135e30 con 0x7f35a00634b0 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a01360f0 con 0x7f35a0142540 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f359ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a00630e0 0x7f35a0134190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f359ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a00630e0 0x7f35a0134190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38918/0 (socket says 172.21.15.121:38918) 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f359ffff700 1 -- 172.21.15.121:0/287468466 learned_addr learned my addr 172.21.15.121:0/287468466 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a4fa2700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a0142540 0x7f35a00a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f359f7fe700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35a00634b0 0x7f35a0135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a4fa2700 1 -- 172.21.15.121:0/287468466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35a00634b0 msgr2=0x7f35a0135320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a4fa2700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35a00634b0 0x7f35a0135320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a4fa2700 1 -- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a00630e0 msgr2=0x7f35a0134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a4fa2700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a00630e0 0x7f35a0134190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a4fa2700 1 -- 172.21.15.121:0/287468466 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940093c0 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a4fa2700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a0142540 0x7f35a00a1770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3598009e50 tx=0x7f359800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f3587fff700 1 -- 172.21.15.121:0/287468466 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359800caa0 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f3587fff700 1 -- 172.21.15.121:0/287468466 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359800f040 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a006ccb0 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f3587fff700 1 -- 172.21.15.121:0/287468466 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359800ce50 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.479+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a0134890 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.480+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a006cf80 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.481+0000 7f3587fff700 1 -- 172.21.15.121:0/287468466 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f3598014690 con 0x7f35a0142540 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.481+0000 7f3587fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.481+0000 7f3587fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:22.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.481+0000 7f3587fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.481+0000 7f3587fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.481+0000 7f3587fff700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f358c098780 0x7f358c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.482+0000 7f3587fff700 1 -- 172.21.15.121:0/287468466 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f35980c10a0 con 0x7f35a0142540 2023-09-29T23:06:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.482+0000 7f359ffff700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f358c098780 0x7f358c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.482+0000 7f359ffff700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f358c098780 0x7f358c09ac30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3594009160 tx=0x7f3594016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.486+0000 7f3587fff700 1 -- 172.21.15.121:0/287468466 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35a006cf80 con 0x7f35a0142540 2023-09-29T23:06:22.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.665+0000 7f35a6a05700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-29T23:06:22.665 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.665+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 --> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35a006cf80 con 0x7f358c098780 2023-09-29T23:06:22.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.671+0000 7f3587fff700 1 -- 172.21.15.121:0/287468466 <== mgr.4107 v2:172.21.15.121:6808/113227 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259742 (secure 0 0 0) 0x7f35a006cf80 con 0x7f358c098780 2023-09-29T23:06:22.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.671+0000 7f3587fff700 20 mgrc handle_command_reply tid 0 r 0 2023-09-29T23:06:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:22.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 10 mgrc shutdown 2023-09-29T23:06:22.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f358c098780 msgr2=0x7f358c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f358c098780 0x7f358c09ac30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3594009160 tx=0x7f3594016000 comp rx=0 tx=0).stop 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a0142540 msgr2=0x7f35a00a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a0142540 0x7f35a00a1770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3598009e50 tx=0x7f359800b040 comp rx=0 tx=0).stop 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 shutdown_connections 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f358c098780 0x7f358c09ac30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35a0142540 0x7f35a00a1770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a00630e0 0x7f35a0134190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 --2- 172.21.15.121:0/287468466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35a00634b0 0x7f35a0135320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 >> 172.21.15.121:0/287468466 conn(0x7f35a00a5bc0 msgr2=0x7f35a0136a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 shutdown_connections 2023-09-29T23:06:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.674+0000 7f35a6a05700 1 -- 172.21.15.121:0/287468466 wait complete. 2023-09-29T23:06:22.678 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2023-09-29T23:06:22.686 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-09-29T23:06:21.486549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":33744,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":33088,"kb_avail":377453616,"statfs":{"total":386547056640,"available":386512502784,"internally_reserved":0,"allocated":638976,"data_stored":1344008,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":33882112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.622899"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869376+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868073+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.872445+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866503+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869137+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866518+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869342+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871195+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866410+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868480+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868259+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868869+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868303+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869355+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868341+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.872531+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868258+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871245+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871420+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:11.325849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383404+0000","last_change":"2023-09-29T23:06:13.639723+0000","last_active":"2023-09-29T23:06:14.383404+0000","last_peered":"2023-09-29T23:06:14.383404+0000","last_clean":"2023-09-29T23:06:14.383404+0000","last_became_active":"2023-09-29T23:06:13.639406+0000","last_became_peered":"2023-09-29T23:06:13.639406+0000","last_unstale":"2023-09-29T23:06:14.383404+0000","last_undegraded":"2023-09-29T23:06:14.383404+0000","last_fullsized":"2023-09-29T23:06:14.383404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:07:23.007465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868356+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380735+0000","last_change":"2023-09-29T23:06:13.380282+0000","last_active":"2023-09-29T23:06:14.380735+0000","last_peered":"2023-09-29T23:06:14.380735+0000","last_clean":"2023-09-29T23:06:14.380735+0000","last_became_active":"2023-09-29T23:06:13.379948+0000","last_became_peered":"2023-09-29T23:06:13.379948+0000","last_unstale":"2023-09-29T23:06:14.380735+0000","last_undegraded":"2023-09-29T23:06:14.380735+0000","last_fullsized":"2023-09-29T23:06:14.380735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:02.759420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869077+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.930970+0000","last_change":"2023-09-29T23:06:13.639753+0000","last_active":"2023-09-29T23:06:14.930970+0000","last_peered":"2023-09-29T23:06:14.930970+0000","last_clean":"2023-09-29T23:06:14.930970+0000","last_became_active":"2023-09-29T23:06:13.639436+0000","last_became_peered":"2023-09-29T23:06:13.639436+0000","last_unstale":"2023-09-29T23:06:14.930970+0000","last_undegraded":"2023-09-29T23:06:14.930970+0000","last_fullsized":"2023-09-29T23:06:14.930970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:09:57.060849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866385+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380753+0000","last_change":"2023-09-29T23:06:13.380967+0000","last_active":"2023-09-29T23:06:14.380753+0000","last_peered":"2023-09-29T23:06:14.380753+0000","last_clean":"2023-09-29T23:06:14.380753+0000","last_became_active":"2023-09-29T23:06:13.379890+0000","last_became_peered":"2023-09-29T23:06:13.379890+0000","last_unstale":"2023-09-29T23:06:14.380753+0000","last_undegraded":"2023-09-29T23:06:14.380753+0000","last_fullsized":"2023-09-29T23:06:14.380753+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:46.738824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'117","reported_seq":163,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383325+0000","last_change":"2023-09-29T23:06:09.717533+0000","last_active":"2023-09-29T23:06:14.383325+0000","last_peered":"2023-09-29T23:06:14.383325+0000","last_clean":"2023-09-29T23:06:14.383325+0000","last_became_active":"2023-09-29T23:06:09.716536+0000","last_became_peered":"2023-09-29T23:06:09.716536+0000","last_unstale":"2023-09-29T23:06:14.383325+0000","last_undegraded":"2023-09-29T23:06:14.383325+0000","last_fullsized":"2023-09-29T23:06:14.383325+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_clean_scrub_stamp":"2023-09-29T23:06:08.290533+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:44:31.128783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871160+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380737+0000","last_change":"2023-09-29T23:06:13.380209+0000","last_active":"2023-09-29T23:06:14.380737+0000","last_peered":"2023-09-29T23:06:14.380737+0000","last_clean":"2023-09-29T23:06:14.380737+0000","last_became_active":"2023-09-29T23:06:13.379845+0000","last_became_peered":"2023-09-29T23:06:13.379845+0000","last_unstale":"2023-09-29T23:06:14.380737+0000","last_undegraded":"2023-09-29T23:06:14.380737+0000","last_fullsized":"2023-09-29T23:06:14.380737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866580+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380734+0000","last_change":"2023-09-29T23:06:13.380613+0000","last_active":"2023-09-29T23:06:14.380734+0000","last_peered":"2023-09-29T23:06:14.380734+0000","last_clean":"2023-09-29T23:06:14.380734+0000","last_became_active":"2023-09-29T23:06:13.380347+0000","last_became_peered":"2023-09-29T23:06:13.380347+0000","last_unstale":"2023-09-29T23:06:14.380734+0000","last_undegraded":"2023-09-29T23:06:14.380734+0000","last_fullsized":"2023-09-29T23:06:14.380734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:35.568211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866389+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871592+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383367+0000","last_change":"2023-09-29T23:06:13.379718+0000","last_active":"2023-09-29T23:06:14.383367+0000","last_peered":"2023-09-29T23:06:14.383367+0000","last_clean":"2023-09-29T23:06:14.383367+0000","last_became_active":"2023-09-29T23:06:13.379450+0000","last_became_peered":"2023-09-29T23:06:13.379450+0000","last_unstale":"2023-09-29T23:06:14.383367+0000","last_undegraded":"2023-09-29T23:06:14.383367+0000","last_fullsized":"2023-09-29T23:06:14.383367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380884+0000","last_change":"2023-09-29T23:06:13.382829+0000","last_active":"2023-09-29T23:06:14.380884+0000","last_peered":"2023-09-29T23:06:14.380884+0000","last_clean":"2023-09-29T23:06:14.380884+0000","last_became_active":"2023-09-29T23:06:13.382590+0000","last_became_peered":"2023-09-29T23:06:13.382590+0000","last_unstale":"2023-09-29T23:06:14.380884+0000","last_undegraded":"2023-09-29T23:06:14.380884+0000","last_fullsized":"2023-09-29T23:06:14.380884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:06.822425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866864+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.873104+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868072+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.870113+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:29:04.705048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871685+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868072+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8136,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363704,"statfs":{"total":96636764160,"available":96628432896,"internally_reserved":0,"allocated":131072,"data_stored":40818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8704,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94363136,"statfs":{"total":96636764160,"available":96627851264,"internally_reserved":0,"allocated":188416,"data_stored":631186,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":131072,"data_stored":40818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8704,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94363136,"statfs":{"total":96636764160,"available":96627851264,"internally_reserved":0,"allocated":188416,"data_stored":631186,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:06:22.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:06:22.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.845+0000 7fd1b13c4700 1 Processor -- start 2023-09-29T23:06:22.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.845+0000 7fd1b13c4700 1 -- start start 2023-09-29T23:06:22.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1b13c4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a91a0 0x7fd1ac0a2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1b13c4700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd1ac0a0c40 0x7fd1ac0a1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1b13c4700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1ac0a2650 con 0x7fd1ac068240 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1b13c4700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ac0a2b40 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1b13c4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0a2880 con 0x7fd1ac0a91a0 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1aa7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a91a0 0x7fd1ac0a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1aaffd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd1ac068240 0x7fd1ac13fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53962/0 (socket says 172.21.15.121:53962) 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1aaffd700 1 -- 172.21.15.121:0/3065596077 learned_addr learned my addr 172.21.15.121:0/3065596077 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1ab7fe700 1 --2- 172.21.15.121:0/3065596077 >> v2:172.21.15.123:3300/0 conn(0x7fd1ac0a0c40 0x7fd1ac0a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1aa7fc700 1 -- 172.21.15.121:0/3065596077 >> v2:172.21.15.123:3300/0 conn(0x7fd1ac0a0c40 msgr2=0x7fd1ac0a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1aa7fc700 1 --2- 172.21.15.121:0/3065596077 >> v2:172.21.15.123:3300/0 conn(0x7fd1ac0a0c40 0x7fd1ac0a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1aa7fc700 1 -- 172.21.15.121:0/3065596077 >> v1:172.21.15.121:6789/0 conn(0x7fd1ac068240 legacy=0x7fd1ac13fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1a9ffb700 1 -- 172.21.15.121:0/3065596077 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181120906 0 0) 0x7fd1ac0a2650 con 0x7fd1ac068240 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1aa7fc700 1 -- 172.21.15.121:0/3065596077 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ac068610 con 0x7fd1ac0a91a0 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.846+0000 7fd1ab7fe700 1 --2- 172.21.15.121:0/3065596077 >> v2:172.21.15.123:3300/0 conn(0x7fd1ac0a0c40 0x7fd1ac0a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1aa7fc700 1 --2- 172.21.15.121:0/3065596077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a91a0 0x7fd1ac0a2110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0098e0 tx=0x7fd19c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c385612fc570647 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1a9ffb700 1 -- 172.21.15.121:0/3065596077 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c009e80 con 0x7fd1ac0a91a0 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1a9ffb700 1 -- 172.21.15.121:0/3065596077 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00b860 con 0x7fd1ac0a91a0 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1a9ffb700 1 -- 172.21.15.121:0/3065596077 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00bbe0 con 0x7fd1ac0a91a0 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 -- 172.21.15.121:0/3065596077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a91a0 msgr2=0x7fd1ac0a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/3065596077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a91a0 0x7fd1ac0a2110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0098e0 tx=0x7fd19c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 -- 172.21.15.121:0/3065596077 shutdown_connections 2023-09-29T23:06:22.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/3065596077 >> v2:172.21.15.123:3300/0 conn(0x7fd1ac0a0c40 0x7fd1ac0a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/3065596077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a91a0 0x7fd1ac0a2110 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 -- 172.21.15.121:0/3065596077 >> 172.21.15.121:0/3065596077 conn(0x7fd1ac06f6a0 msgr2=0x7fd1ac067970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:22.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 -- 172.21.15.121:0/3065596077 shutdown_connections 2023-09-29T23:06:22.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.847+0000 7fd1b13c4700 1 -- 172.21.15.121:0/3065596077 wait complete. 2023-09-29T23:06:22.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.848+0000 7fd1b13c4700 1 Processor -- start 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.848+0000 7fd1b13c4700 1 -- start start 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1b13c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac068240 0x7fd1ac143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1b13c4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a0c40 0x7fd1ac1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1b13c4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1ac0a91a0 0x7fd1ac13eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1b13c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac143760 con 0x7fd1ac068240 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1b13c4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac153f40 con 0x7fd1ac0a91a0 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1b13c4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac154200 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a0c40 0x7fd1ac1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac068240 0x7fd1ac143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a0c40 0x7fd1ac1536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:35016/0 (socket says 172.21.15.121:35016) 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 -- 172.21.15.121:0/1939049704 learned_addr learned my addr 172.21.15.121:0/1939049704 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1ab7fe700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1ac0a91a0 0x7fd1ac13eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 -- 172.21.15.121:0/1939049704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1ac0a91a0 msgr2=0x7fd1ac13eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1ac0a91a0 0x7fd1ac13eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 -- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac068240 msgr2=0x7fd1ac143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac068240 0x7fd1ac143ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aa7fc700 1 -- 172.21.15.121:0/1939049704 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c009420 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.849+0000 7fd1aaffd700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac068240 0x7fd1ac143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.850+0000 7fd1aa7fc700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a0c40 0x7fd1ac1536f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac069f90 tx=0x7fd19c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.850+0000 7fd192ffd700 1 -- 172.21.15.121:0/1939049704 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c000cc0 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.850+0000 7fd192ffd700 1 -- 172.21.15.121:0/1939049704 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00b9d0 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.850+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ac1544c0 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.850+0000 7fd192ffd700 1 -- 172.21.15.121:0/1939049704 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c025670 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.850+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ac153cb0 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.851+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ac069080 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd192ffd700 1 -- 172.21.15.121:0/1939049704 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd19c023070 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd192ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd192ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd192ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd192ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd192ffd700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd198098780 0x7fd19809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd1aaffd700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd198098780 0x7fd19809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:22.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.852+0000 7fd192ffd700 1 -- 172.21.15.121:0/1939049704 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fd19c0c1870 con 0x7fd1ac0a0c40 2023-09-29T23:06:22.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.853+0000 7fd1aaffd700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd198098780 0x7fd19809ac30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0007fd0 tx=0x7fd1a0007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:22.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:22.855+0000 7fd192ffd700 1 -- 172.21.15.121:0/1939049704 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd19c08efb0 con 0x7fd1ac0a0c40 2023-09-29T23:06:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.029+0000 7fd1b13c4700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-29T23:06:23.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.029+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 --> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1ac071660 con 0x7fd198098780 2023-09-29T23:06:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.035+0000 7fd192ffd700 1 -- 172.21.15.121:0/1939049704 <== mgr.4107 v2:172.21.15.121:6808/113227 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259742 (secure 0 0 0) 0x7fd1ac071660 con 0x7fd198098780 2023-09-29T23:06:23.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.035+0000 7fd192ffd700 20 mgrc handle_command_reply tid 0 r 0 2023-09-29T23:06:23.038 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 10 mgrc shutdown 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd198098780 msgr2=0x7fd19809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd198098780 0x7fd19809ac30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0007fd0 tx=0x7fd1a0007d30 comp rx=0 tx=0).stop 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a0c40 msgr2=0x7fd1ac1536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a0c40 0x7fd1ac1536f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac069f90 tx=0x7fd19c017000 comp rx=0 tx=0).stop 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 shutdown_connections 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd198098780 0x7fd19809ac30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1ac0a91a0 0x7fd1ac13eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac068240 0x7fd1ac143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 --2- 172.21.15.121:0/1939049704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac0a0c40 0x7fd1ac1536f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 >> 172.21.15.121:0/1939049704 conn(0x7fd1ac06f6a0 msgr2=0x7fd1ac13fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 shutdown_connections 2023-09-29T23:06:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.038+0000 7fd1b13c4700 1 -- 172.21.15.121:0/1939049704 wait complete. 2023-09-29T23:06:23.042 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2023-09-29T23:06:23.050 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-09-29T23:06:21.486549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":33744,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":33088,"kb_avail":377453616,"statfs":{"total":386547056640,"available":386512502784,"internally_reserved":0,"allocated":638976,"data_stored":1344008,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":33882112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.622899"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869376+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868073+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.872445+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866503+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869137+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866518+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869342+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871195+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866410+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868480+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868259+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868869+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868303+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869355+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868341+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.872531+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868258+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871245+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871420+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:11.325849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383404+0000","last_change":"2023-09-29T23:06:13.639723+0000","last_active":"2023-09-29T23:06:14.383404+0000","last_peered":"2023-09-29T23:06:14.383404+0000","last_clean":"2023-09-29T23:06:14.383404+0000","last_became_active":"2023-09-29T23:06:13.639406+0000","last_became_peered":"2023-09-29T23:06:13.639406+0000","last_unstale":"2023-09-29T23:06:14.383404+0000","last_undegraded":"2023-09-29T23:06:14.383404+0000","last_fullsized":"2023-09-29T23:06:14.383404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:07:23.007465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868356+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380735+0000","last_change":"2023-09-29T23:06:13.380282+0000","last_active":"2023-09-29T23:06:14.380735+0000","last_peered":"2023-09-29T23:06:14.380735+0000","last_clean":"2023-09-29T23:06:14.380735+0000","last_became_active":"2023-09-29T23:06:13.379948+0000","last_became_peered":"2023-09-29T23:06:13.379948+0000","last_unstale":"2023-09-29T23:06:14.380735+0000","last_undegraded":"2023-09-29T23:06:14.380735+0000","last_fullsized":"2023-09-29T23:06:14.380735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:02.759420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.869077+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.930970+0000","last_change":"2023-09-29T23:06:13.639753+0000","last_active":"2023-09-29T23:06:14.930970+0000","last_peered":"2023-09-29T23:06:14.930970+0000","last_clean":"2023-09-29T23:06:14.930970+0000","last_became_active":"2023-09-29T23:06:13.639436+0000","last_became_peered":"2023-09-29T23:06:13.639436+0000","last_unstale":"2023-09-29T23:06:14.930970+0000","last_undegraded":"2023-09-29T23:06:14.930970+0000","last_fullsized":"2023-09-29T23:06:14.930970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:09:57.060849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866385+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380753+0000","last_change":"2023-09-29T23:06:13.380967+0000","last_active":"2023-09-29T23:06:14.380753+0000","last_peered":"2023-09-29T23:06:14.380753+0000","last_clean":"2023-09-29T23:06:14.380753+0000","last_became_active":"2023-09-29T23:06:13.379890+0000","last_became_peered":"2023-09-29T23:06:13.379890+0000","last_unstale":"2023-09-29T23:06:14.380753+0000","last_undegraded":"2023-09-29T23:06:14.380753+0000","last_fullsized":"2023-09-29T23:06:14.380753+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:46.738824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'117","reported_seq":163,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383325+0000","last_change":"2023-09-29T23:06:09.717533+0000","last_active":"2023-09-29T23:06:14.383325+0000","last_peered":"2023-09-29T23:06:14.383325+0000","last_clean":"2023-09-29T23:06:14.383325+0000","last_became_active":"2023-09-29T23:06:09.716536+0000","last_became_peered":"2023-09-29T23:06:09.716536+0000","last_unstale":"2023-09-29T23:06:14.383325+0000","last_undegraded":"2023-09-29T23:06:14.383325+0000","last_fullsized":"2023-09-29T23:06:14.383325+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_clean_scrub_stamp":"2023-09-29T23:06:08.290533+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:44:31.128783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871160+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380737+0000","last_change":"2023-09-29T23:06:13.380209+0000","last_active":"2023-09-29T23:06:14.380737+0000","last_peered":"2023-09-29T23:06:14.380737+0000","last_clean":"2023-09-29T23:06:14.380737+0000","last_became_active":"2023-09-29T23:06:13.379845+0000","last_became_peered":"2023-09-29T23:06:13.379845+0000","last_unstale":"2023-09-29T23:06:14.380737+0000","last_undegraded":"2023-09-29T23:06:14.380737+0000","last_fullsized":"2023-09-29T23:06:14.380737+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866580+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380734+0000","last_change":"2023-09-29T23:06:13.380613+0000","last_active":"2023-09-29T23:06:14.380734+0000","last_peered":"2023-09-29T23:06:14.380734+0000","last_clean":"2023-09-29T23:06:14.380734+0000","last_became_active":"2023-09-29T23:06:13.380347+0000","last_became_peered":"2023-09-29T23:06:13.380347+0000","last_unstale":"2023-09-29T23:06:14.380734+0000","last_undegraded":"2023-09-29T23:06:14.380734+0000","last_fullsized":"2023-09-29T23:06:14.380734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:35.568211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866389+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871592+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.383367+0000","last_change":"2023-09-29T23:06:13.379718+0000","last_active":"2023-09-29T23:06:14.383367+0000","last_peered":"2023-09-29T23:06:14.383367+0000","last_clean":"2023-09-29T23:06:14.383367+0000","last_became_active":"2023-09-29T23:06:13.379450+0000","last_became_peered":"2023-09-29T23:06:13.379450+0000","last_unstale":"2023-09-29T23:06:14.383367+0000","last_undegraded":"2023-09-29T23:06:14.383367+0000","last_fullsized":"2023-09-29T23:06:14.383367+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-29T23:06:14.380884+0000","last_change":"2023-09-29T23:06:13.382829+0000","last_active":"2023-09-29T23:06:14.380884+0000","last_peered":"2023-09-29T23:06:14.380884+0000","last_clean":"2023-09-29T23:06:14.380884+0000","last_became_active":"2023-09-29T23:06:13.382590+0000","last_became_peered":"2023-09-29T23:06:13.382590+0000","last_unstale":"2023-09-29T23:06:14.380884+0000","last_undegraded":"2023-09-29T23:06:14.380884+0000","last_fullsized":"2023-09-29T23:06:14.380884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:06.822425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.866864+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.873104+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868072+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.870113+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:29:04.705048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.871685+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-29T23:06:16.860955+0000","last_change":"2023-09-29T23:06:16.868072+0000","last_active":"2023-09-29T23:06:16.860955+0000","last_peered":"2023-09-29T23:06:16.860955+0000","last_clean":"2023-09-29T23:06:16.860955+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:16.860955+0000","last_undegraded":"2023-09-29T23:06:16.860955+0000","last_fullsized":"2023-09-29T23:06:16.860955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-09-29T23:06:17.862754+0000","last_change":"2023-09-29T23:06:17.862754+0000","last_active":"2023-09-29T23:06:17.862754+0000","last_peered":"2023-09-29T23:06:17.862754+0000","last_clean":"2023-09-29T23:06:17.862754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T23:06:17.862754+0000","last_undegraded":"2023-09-29T23:06:17.862754+0000","last_fullsized":"2023-09-29T23:06:17.862754+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8136,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363704,"statfs":{"total":96636764160,"available":96628432896,"internally_reserved":0,"allocated":131072,"data_stored":40818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8704,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94363136,"statfs":{"total":96636764160,"available":96627851264,"internally_reserved":0,"allocated":188416,"data_stored":631186,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8200,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363640,"statfs":{"total":96636764160,"available":96628367360,"internally_reserved":0,"allocated":131072,"data_stored":40818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8704,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8512,"kb_avail":94363136,"statfs":{"total":96636764160,"available":96627851264,"internally_reserved":0,"allocated":188416,"data_stored":631186,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8716288},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:06:23.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3f is not active+clean 2023-09-29T23:06:23.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3e is not active+clean 2023-09-29T23:06:23.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3d is not active+clean 2023-09-29T23:06:23.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3c is not active+clean 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3b is not active+clean 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3a is not active+clean 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.39 is not active+clean 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.38 is not active+clean 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.37 is not active+clean 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.36 is not active+clean 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.35 is not active+clean 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.34 is not active+clean 2023-09-29T23:06:23.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.33 is not active+clean 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.32 is not active+clean 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.31 is not active+clean 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.30 is not active+clean 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2f is not active+clean 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2e is not active+clean 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2d is not active+clean 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2c is not active+clean 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2b is not active+clean 2023-09-29T23:06:23.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2a is not active+clean 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.29 is not active+clean 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.28 is not active+clean 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.27 is not active+clean 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.26 is not active+clean 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.25 is not active+clean 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1a is not active+clean 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.869376+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1d is not active+clean 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.19 is not active+clean 2023-09-29T23:06:23.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868073+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1e is not active+clean 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.18 is not active+clean 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.872445+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1f is not active+clean 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.17 is not active+clean 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.866503+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T08:21:07.120696+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.10 is not active+clean 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.16 is not active+clean 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.869137+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.11 is not active+clean 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.15 is not active+clean 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.866518+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.12 is not active+clean 2023-09-29T23:06:23.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.14 is not active+clean 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.869342+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.13 is not active+clean 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.13 is not active+clean 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.871195+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T04:33:51.242247+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.14 is not active+clean 2023-09-29T23:06:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.12 is not active+clean 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.866410+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.15 is not active+clean 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.11 is not active+clean 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868480+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.16 is not active+clean 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.10 is not active+clean 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868259+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.17 is not active+clean 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.f is not active+clean 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868869+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.8 is not active+clean 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.e is not active+clean 2023-09-29T23:06:23.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868303+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.9 is not active+clean 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.d is not active+clean 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.869355+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.a is not active+clean 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.c is not active+clean 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868341+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.069 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.b is not active+clean 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.b is not active+clean 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.872531+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.c is not active+clean 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.a is not active+clean 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868258+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.d is not active+clean 2023-09-29T23:06:23.070 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.9 is not active+clean 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.871245+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T00:53:40.071634+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.e is not active+clean 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.8 is not active+clean 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.871420+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T04:44:11.325849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.f is not active+clean 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.071 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.5 is not active+clean 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868356+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2 is not active+clean 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3 is not active+clean 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.869077+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.4 is not active+clean 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.0 is not active+clean 2023-09-29T23:06:23.072 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.866385+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.7 is not active+clean 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2 is not active+clean 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.871160+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.5 is not active+clean 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1 is not active+clean 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.866580+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.073 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.6 is not active+clean 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.4 is not active+clean 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.866389+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3 is not active+clean 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.7 is not active+clean 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.871592+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.0 is not active+clean 2023-09-29T23:06:23.074 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.6 is not active+clean 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.866864+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T07:53:57.983726+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1 is not active+clean 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1b is not active+clean 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.873104+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1c is not active+clean 2023-09-29T23:06:23.075 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1c is not active+clean 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868072+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1b is not active+clean 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1d is not active+clean 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.870113+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T03:29:04.705048+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1a is not active+clean 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1e is not active+clean 2023-09-29T23:06:23.076 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.871685+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T01:07:42.460527+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.19 is not active+clean 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1f is not active+clean 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-29T23:06:16.860955+0000', 'last_change': '2023-09-29T23:06:16.868072+0000', 'last_active': '2023-09-29T23:06:16.860955+0000', 'last_peered': '2023-09-29T23:06:16.860955+0000', 'last_clean': '2023-09-29T23:06:16.860955+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:16.860955+0000', 'last_undegraded': '2023-09-29T23:06:16.860955+0000', 'last_fullsized': '2023-09-29T23:06:16.860955+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:16.860955+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.18 is not active+clean 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.20 is not active+clean 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.077 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.21 is not active+clean 2023-09-29T23:06:23.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.22 is not active+clean 2023-09-29T23:06:23.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.23 is not active+clean 2023-09-29T23:06:23.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.24 is not active+clean 2023-09-29T23:06:23.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-09-29T23:06:17.862754+0000', 'last_change': '2023-09-29T23:06:17.862754+0000', 'last_active': '2023-09-29T23:06:17.862754+0000', 'last_peered': '2023-09-29T23:06:17.862754+0000', 'last_clean': '2023-09-29T23:06:17.862754+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T23:06:17.862754+0000', 'last_undegraded': '2023-09-29T23:06:17.862754+0000', 'last_fullsized': '2023-09-29T23:06:17.862754+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'last_clean_scrub_stamp': '2023-09-29T23:06:17.862754+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-09-29T23:06:23.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-09-29T23:06:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.206+0000 7f1d4f935700 1 Processor -- start 2023-09-29T23:06:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4f935700 1 -- start start 2023-09-29T23:06:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4f935700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d480a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4f935700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d4809b770 0x7f1d480630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4f935700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1d480b3cf0 0x7f1d48143df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4f935700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d48141810 con 0x7f1d4809b770 2023-09-29T23:06:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4f935700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4809a7f0 con 0x7f1d480b3cf0 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4f935700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d480a4df0 con 0x7f1d48141440 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4ced0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d4809b770 0x7f1d480630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4d6d1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d480a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4d6d1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d480a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:35020/0 (socket says 172.21.15.121:35020) 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4ced0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d4809b770 0x7f1d480630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38940/0 (socket says 172.21.15.121:38940) 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.207+0000 7f1d4d6d1700 1 -- 172.21.15.121:0/67608336 learned_addr learned my addr 172.21.15.121:0/67608336 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4ded2700 1 --2- 172.21.15.121:0/67608336 >> v2:172.21.15.123:3300/0 conn(0x7f1d480b3cf0 0x7f1d48143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4d6d1700 1 -- 172.21.15.121:0/67608336 >> v2:172.21.15.123:3300/0 conn(0x7f1d480b3cf0 msgr2=0x7f1d48143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:23.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4d6d1700 1 --2- 172.21.15.121:0/67608336 >> v2:172.21.15.123:3300/0 conn(0x7f1d480b3cf0 0x7f1d48143df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4d6d1700 1 -- 172.21.15.121:0/67608336 >> v2:172.21.15.121:3300/0 conn(0x7f1d4809b770 msgr2=0x7f1d480630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4d6d1700 1 --2- 172.21.15.121:0/67608336 >> v2:172.21.15.121:3300/0 conn(0x7f1d4809b770 0x7f1d480630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4d6d1700 1 -- 172.21.15.121:0/67608336 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d480a50b0 con 0x7f1d48141440 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4ced0700 1 --2- 172.21.15.121:0/67608336 >> v2:172.21.15.121:3300/0 conn(0x7f1d4809b770 0x7f1d480630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4ded2700 1 --2- 172.21.15.121:0/67608336 >> v2:172.21.15.123:3300/0 conn(0x7f1d480b3cf0 0x7f1d48143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4d6d1700 1 --2- 172.21.15.121:0/67608336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d480a45a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1d4806b1b0 tx=0x7f1d38009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2cd5cd8c5e2a653 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d3ffff700 1 -- 172.21.15.121:0/67608336 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38005c10 con 0x7f1d48141440 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d3ffff700 1 -- 172.21.15.121:0/67608336 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d38005d70 con 0x7f1d48141440 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d3ffff700 1 -- 172.21.15.121:0/67608336 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38006280 con 0x7f1d48141440 2023-09-29T23:06:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4f935700 1 -- 172.21.15.121:0/67608336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 msgr2=0x7f1d480a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4f935700 1 --2- 172.21.15.121:0/67608336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d480a45a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1d4806b1b0 tx=0x7f1d38009220 comp rx=0 tx=0).stop 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.208+0000 7f1d4f935700 1 -- 172.21.15.121:0/67608336 shutdown_connections 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.209+0000 7f1d4f935700 1 --2- 172.21.15.121:0/67608336 >> v2:172.21.15.123:3300/0 conn(0x7f1d480b3cf0 0x7f1d48143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.209+0000 7f1d4f935700 1 --2- 172.21.15.121:0/67608336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d480a45a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.209+0000 7f1d4f935700 1 --2- 172.21.15.121:0/67608336 >> v2:172.21.15.121:3300/0 conn(0x7f1d4809b770 0x7f1d480630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.209+0000 7f1d4f935700 1 -- 172.21.15.121:0/67608336 >> 172.21.15.121:0/67608336 conn(0x7f1d48069e90 msgr2=0x7f1d480b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.209+0000 7f1d4f935700 1 -- 172.21.15.121:0/67608336 shutdown_connections 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.209+0000 7f1d4f935700 1 -- 172.21.15.121:0/67608336 wait complete. 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.209+0000 7f1d4f935700 1 Processor -- start 2023-09-29T23:06:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4f935700 1 -- start start 2023-09-29T23:06:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4f935700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d4809b770 0x7f1d4806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4f935700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d480b3cf0 0x7f1d4806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4f935700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d4806c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4f935700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4806f0d0 con 0x7f1d4809b770 2023-09-29T23:06:23.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4f935700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4806f230 con 0x7f1d480b3cf0 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4f935700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4806ccd0 con 0x7f1d48141440 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4d6d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d4809b770 0x7f1d4806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4ded2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d4806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4d6d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d4809b770 0x7f1d4806d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38954/0 (socket says 172.21.15.121:38954) 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4d6d1700 1 -- 172.21.15.121:0/4161849913 learned_addr learned my addr 172.21.15.121:0/4161849913 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.210+0000 7f1d4ced0700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d480b3cf0 0x7f1d4806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4ded2700 1 -- 172.21.15.121:0/4161849913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d480b3cf0 msgr2=0x7f1d4806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4ded2700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d480b3cf0 0x7f1d4806eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4ded2700 1 -- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d4809b770 msgr2=0x7f1d4806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4ded2700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d4809b770 0x7f1d4806d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4ded2700 1 -- 172.21.15.121:0/4161849913 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44009420 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4d6d1700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d4809b770 0x7f1d4806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4ded2700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d4806c790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000cf30 tx=0x7f1d40007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d3d7fa700 1 -- 172.21.15.121:0/4161849913 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4000ae90 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d3d7fa700 1 -- 172.21.15.121:0/4161849913 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d40015440 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d38009480 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d3d7fa700 1 -- 172.21.15.121:0/4161849913 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40015800 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.211+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4806d1a0 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.212+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4806ab60 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.213+0000 7f1d3d7fa700 1 -- 172.21.15.121:0/4161849913 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f1d4000d440 con 0x7f1d48141440 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.213+0000 7f1d3d7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.213+0000 7f1d3d7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.213+0000 7f1d3d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.213+0000 7f1d3d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.213+0000 7f1d3d7fa700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d2c098780 0x7f1d2c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.213+0000 7f1d4d6d1700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d2c098780 0x7f1d2c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.214+0000 7f1d3d7fa700 1 -- 172.21.15.121:0/4161849913 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f1d400c18b0 con 0x7f1d48141440 2023-09-29T23:06:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.214+0000 7f1d4d6d1700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d2c098780 0x7f1d2c09ac30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1d38007fd0 tx=0x7f1d38016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:23.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.217+0000 7f1d3d7fa700 1 -- 172.21.15.121:0/4161849913 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d4008eec0 con 0x7f1d48141440 2023-09-29T23:06:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.471+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1d48071660 con 0x7f1d48141440 2023-09-29T23:06:23.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.471+0000 7f1d3d7fa700 1 -- 172.21.15.121:0/4161849913 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1289 (secure 0 0 0) 0x7f1d4001a020 con 0x7f1d48141440 2023-09-29T23:06:23.471 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:23.471 INFO:teuthology.orchestra.run.smithi121.stdout:{"fsid":"f139d264-5213-413d-9960-69a6d67478bb","health":{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":27,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1696028767,"num_in_osds":4,"osd_in_since":1696028757,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":9}],"num_pgs":105,"num_pools":4,"num_objects":2,"data_bytes":590368,"bytes_used":34553856,"bytes_avail":386512502784,"bytes_total":386547056640,"unknown_pgs_ratio":0.60952383279800415,"inactive_pgs_ratio":0.30476191639900208},"fsmap":{"epoch":8,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":4240}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-09-29T23:05:56.091545+0000","services":{}},"progress_events":{}} 2023-09-29T23:06:23.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 10 mgrc shutdown 2023-09-29T23:06:23.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d2c098780 msgr2=0x7f1d2c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d2c098780 0x7f1d2c09ac30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1d38007fd0 tx=0x7f1d38016000 comp rx=0 tx=0).stop 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 msgr2=0x7f1d4806c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d4806c790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000cf30 tx=0x7f1d40007840 comp rx=0 tx=0).stop 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 shutdown_connections 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f1d2c098780 0x7f1d2c09ac30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d48141440 0x7f1d4806c790 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d4809b770 0x7f1d4806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 --2- 172.21.15.121:0/4161849913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d480b3cf0 0x7f1d4806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 >> 172.21.15.121:0/4161849913 conn(0x7f1d48069e90 msgr2=0x7f1d48140590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 shutdown_connections 2023-09-29T23:06:23.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.473+0000 7f1d4f935700 1 -- 172.21.15.121:0/4161849913 wait complete. 2023-09-29T23:06:23.485 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-09-29T23:06:23.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:06:23.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.601+0000 7f21c886a700 1 Processor -- start 2023-09-29T23:06:23.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.601+0000 7f21c886a700 1 -- start start 2023-09-29T23:06:23.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c886a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f21c4067690 0x7f21c406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c886a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21c40a5800 0x7f21c4069a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c886a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c4067a60 con 0x7f21c40a5c40 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c886a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c406dfc0 con 0x7f21c40a5800 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c886a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c406e580 con 0x7f21c4067690 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21c40a5800 0x7f21c4069a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21c40a5800 0x7f21c4069a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38956/0 (socket says 172.21.15.121:38956) 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 -- 172.21.15.121:0/2414292259 learned_addr learned my addr 172.21.15.121:0/2414292259 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c259c700 1 --2- 172.21.15.121:0/2414292259 >> v2:172.21.15.123:3300/0 conn(0x7f21c4067690 0x7f21c406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 -- 172.21.15.121:0/2414292259 >> v2:172.21.15.123:3300/0 conn(0x7f21c4067690 msgr2=0x7f21c406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 --2- 172.21.15.121:0/2414292259 >> v2:172.21.15.123:3300/0 conn(0x7f21c4067690 0x7f21c406d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 -- 172.21.15.121:0/2414292259 >> v1:172.21.15.123:6789/0 conn(0x7f21c40a5c40 legacy=0x7f21c4143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:23.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 -- 172.21.15.121:0/2414292259 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c406a9b0 con 0x7f21c40a5800 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c259c700 1 --2- 172.21.15.121:0/2414292259 >> v2:172.21.15.123:3300/0 conn(0x7f21c4067690 0x7f21c406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c1d9b700 1 --2- 172.21.15.121:0/2414292259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5800 0x7f21c4069a60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21c4069380 tx=0x7f21b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcd1a5db1c829ce server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c159a700 1 -- 172.21.15.121:0/2414292259 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b400b820 con 0x7f21c40a5800 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c159a700 1 -- 172.21.15.121:0/2414292259 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b400b980 con 0x7f21c40a5800 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.602+0000 7f21c159a700 1 -- 172.21.15.121:0/2414292259 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b400bd20 con 0x7f21c40a5800 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 -- 172.21.15.121:0/2414292259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5800 msgr2=0x7f21c4069a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 --2- 172.21.15.121:0/2414292259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5800 0x7f21c4069a60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21c4069380 tx=0x7f21b4009250 comp rx=0 tx=0).stop 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 -- 172.21.15.121:0/2414292259 shutdown_connections 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 --2- 172.21.15.121:0/2414292259 >> v2:172.21.15.123:3300/0 conn(0x7f21c4067690 0x7f21c406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 --2- 172.21.15.121:0/2414292259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5800 0x7f21c4069a60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 -- 172.21.15.121:0/2414292259 >> 172.21.15.121:0/2414292259 conn(0x7f21c4068550 msgr2=0x7f21c4068950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 -- 172.21.15.121:0/2414292259 shutdown_connections 2023-09-29T23:06:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.603+0000 7f21c886a700 1 -- 172.21.15.121:0/2414292259 wait complete. 2023-09-29T23:06:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.604+0000 7f21c886a700 1 Processor -- start 2023-09-29T23:06:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.604+0000 7f21c886a700 1 -- start start 2023-09-29T23:06:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21c4067690 0x7f21c41444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40a5800 0x7f21c40a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5c40 0x7f21c414f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40a3730 con 0x7f21c40a5c40 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c409ec70 con 0x7f21c40a5800 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c409ef30 con 0x7f21c4067690 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21c4067690 0x7f21c41444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5c40 0x7f21c414f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21c4067690 0x7f21c41444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:35032/0 (socket says 172.21.15.121:35032) 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 -- 172.21.15.121:0/1934001822 learned_addr learned my addr 172.21.15.121:0/1934001822 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c1d9b700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40a5800 0x7f21c40a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 -- 172.21.15.121:0/1934001822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40a5800 msgr2=0x7f21c40a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40a5800 0x7f21c40a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 -- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5c40 msgr2=0x7f21c414f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5c40 0x7f21c414f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 -- 172.21.15.121:0/1934001822 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b40094f0 con 0x7f21c4067690 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c1d9b700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40a5800 0x7f21c40a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c2d9d700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5c40 0x7f21c414f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c259c700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21c4067690 0x7f21c41444a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21b800c830 tx=0x7f21b8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21b27fc700 1 -- 172.21.15.121:0/1934001822 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8011600 con 0x7f21c4067690 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21b27fc700 1 -- 172.21.15.121:0/1934001822 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b8011760 con 0x7f21c4067690 2023-09-29T23:06:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c409e8f0 con 0x7f21c4067690 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21b27fc700 1 -- 172.21.15.121:0/1934001822 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8011b20 con 0x7f21c4067690 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.605+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c4142e80 con 0x7f21c4067690 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.606+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c4071660 con 0x7f21c4067690 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21b27fc700 1 -- 172.21.15.121:0/1934001822 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f21b8024480 con 0x7f21c4067690 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21b27fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21b27fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21b27fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21b27fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21b27fc700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21a4098780 0x7f21a409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21c1d9b700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21a4098780 0x7f21a409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:23.611 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.608+0000 7f21b27fc700 1 -- 172.21.15.121:0/1934001822 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f21b80c0870 con 0x7f21c4067690 2023-09-29T23:06:23.612 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.609+0000 7f21c1d9b700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21a4098780 0x7f21a409ac30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21c4069380 tx=0x7f21b4016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:23.612 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.611+0000 7f21b27fc700 1 -- 172.21.15.121:0/1934001822 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21b808dfb0 con 0x7f21c4067690 2023-09-29T23:06:23.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.787+0000 7f21c886a700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-29T23:06:23.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.787+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 --> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21c4143b50 con 0x7f21a4098780 2023-09-29T23:06:23.793 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.794+0000 7f21b27fc700 1 -- 172.21.15.121:0/1934001822 <== mgr.4107 v2:172.21.15.121:6808/113227 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+269438 (secure 0 0 0) 0x7f21c4143b50 con 0x7f21a4098780 2023-09-29T23:06:23.793 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.794+0000 7f21b27fc700 20 mgrc handle_command_reply tid 0 r 0 2023-09-29T23:06:23.797 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:23.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.796+0000 7f21c886a700 10 mgrc shutdown 2023-09-29T23:06:23.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21a4098780 msgr2=0x7f21a409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21a4098780 0x7f21a409ac30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f21c4069380 tx=0x7f21b4016000 comp rx=0 tx=0).stop 2023-09-29T23:06:23.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21c4067690 msgr2=0x7f21c41444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:23.799 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21c4067690 0x7f21c41444a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21b800c830 tx=0x7f21b8008040 comp rx=0 tx=0).stop 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 shutdown_connections 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21a4098780 0x7f21a409ac30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21c40a5c40 0x7f21c414f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21c4067690 0x7f21c41444a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 --2- 172.21.15.121:0/1934001822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40a5800 0x7f21c40a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 >> 172.21.15.121:0/1934001822 conn(0x7f21c4068550 msgr2=0x7f21c40b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 shutdown_connections 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:23.797+0000 7f21c886a700 1 -- 172.21.15.121:0/1934001822 wait complete. 2023-09-29T23:06:23.800 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2023-09-29T23:06:23.808 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-09-29T23:06:23.486852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":35136,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":377452224,"statfs":{"total":386547056640,"available":386511077376,"internally_reserved":0,"allocated":884736,"data_stored":1426840,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001067"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278366+0000","last_change":"2023-09-29T23:06:18.276673+0000","last_active":"2023-09-29T23:06:18.278366+0000","last_peered":"2023-09-29T23:06:18.278366+0000","last_clean":"2023-09-29T23:06:18.278366+0000","last_became_active":"2023-09-29T23:06:18.276312+0000","last_became_peered":"2023-09-29T23:06:18.276312+0000","last_unstale":"2023-09-29T23:06:18.278366+0000","last_undegraded":"2023-09-29T23:06:18.278366+0000","last_fullsized":"2023-09-29T23:06:18.278366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:40.183958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283354+0000","last_change":"2023-09-29T23:06:18.282307+0000","last_active":"2023-09-29T23:06:18.283354+0000","last_peered":"2023-09-29T23:06:18.283354+0000","last_clean":"2023-09-29T23:06:18.283354+0000","last_became_active":"2023-09-29T23:06:18.282064+0000","last_became_peered":"2023-09-29T23:06:18.282064+0000","last_unstale":"2023-09-29T23:06:18.283354+0000","last_undegraded":"2023-09-29T23:06:18.283354+0000","last_fullsized":"2023-09-29T23:06:18.283354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:10:08.336407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283157+0000","last_change":"2023-09-29T23:06:18.281514+0000","last_active":"2023-09-29T23:06:18.283157+0000","last_peered":"2023-09-29T23:06:18.283157+0000","last_clean":"2023-09-29T23:06:18.283157+0000","last_became_active":"2023-09-29T23:06:18.281144+0000","last_became_peered":"2023-09-29T23:06:18.281144+0000","last_unstale":"2023-09-29T23:06:18.283157+0000","last_undegraded":"2023-09-29T23:06:18.283157+0000","last_fullsized":"2023-09-29T23:06:18.283157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290407+0000","last_change":"2023-09-29T23:06:18.287192+0000","last_active":"2023-09-29T23:06:18.290407+0000","last_peered":"2023-09-29T23:06:18.290407+0000","last_clean":"2023-09-29T23:06:18.290407+0000","last_became_active":"2023-09-29T23:06:18.286917+0000","last_became_peered":"2023-09-29T23:06:18.286917+0000","last_unstale":"2023-09-29T23:06:18.290407+0000","last_undegraded":"2023-09-29T23:06:18.290407+0000","last_fullsized":"2023-09-29T23:06:18.290407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292437+0000","last_change":"2023-09-29T23:06:18.289803+0000","last_active":"2023-09-29T23:06:18.292437+0000","last_peered":"2023-09-29T23:06:18.292437+0000","last_clean":"2023-09-29T23:06:18.292437+0000","last_became_active":"2023-09-29T23:06:18.289379+0000","last_became_peered":"2023-09-29T23:06:18.289379+0000","last_unstale":"2023-09-29T23:06:18.292437+0000","last_undegraded":"2023-09-29T23:06:18.292437+0000","last_fullsized":"2023-09-29T23:06:18.292437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287884+0000","last_change":"2023-09-29T23:06:18.285440+0000","last_active":"2023-09-29T23:06:18.287884+0000","last_peered":"2023-09-29T23:06:18.287884+0000","last_clean":"2023-09-29T23:06:18.287884+0000","last_became_active":"2023-09-29T23:06:18.285182+0000","last_became_peered":"2023-09-29T23:06:18.285182+0000","last_unstale":"2023-09-29T23:06:18.287884+0000","last_undegraded":"2023-09-29T23:06:18.287884+0000","last_fullsized":"2023-09-29T23:06:18.287884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:04:37.136451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288766+0000","last_change":"2023-09-29T23:06:18.286007+0000","last_active":"2023-09-29T23:06:18.288766+0000","last_peered":"2023-09-29T23:06:18.288766+0000","last_clean":"2023-09-29T23:06:18.288766+0000","last_became_active":"2023-09-29T23:06:18.284765+0000","last_became_peered":"2023-09-29T23:06:18.284765+0000","last_unstale":"2023-09-29T23:06:18.288766+0000","last_undegraded":"2023-09-29T23:06:18.288766+0000","last_fullsized":"2023-09-29T23:06:18.288766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273887+0000","last_change":"2023-09-29T23:06:18.272963+0000","last_active":"2023-09-29T23:06:18.273887+0000","last_peered":"2023-09-29T23:06:18.273887+0000","last_clean":"2023-09-29T23:06:18.273887+0000","last_became_active":"2023-09-29T23:06:18.272736+0000","last_became_peered":"2023-09-29T23:06:18.272736+0000","last_unstale":"2023-09-29T23:06:18.273887+0000","last_undegraded":"2023-09-29T23:06:18.273887+0000","last_fullsized":"2023-09-29T23:06:18.273887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278347+0000","last_change":"2023-09-29T23:06:18.276308+0000","last_active":"2023-09-29T23:06:18.278347+0000","last_peered":"2023-09-29T23:06:18.278347+0000","last_clean":"2023-09-29T23:06:18.278347+0000","last_became_active":"2023-09-29T23:06:18.276056+0000","last_became_peered":"2023-09-29T23:06:18.276056+0000","last_unstale":"2023-09-29T23:06:18.278347+0000","last_undegraded":"2023-09-29T23:06:18.278347+0000","last_fullsized":"2023-09-29T23:06:18.278347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:36.204055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287088+0000","last_change":"2023-09-29T23:06:18.280125+0000","last_active":"2023-09-29T23:06:18.287088+0000","last_peered":"2023-09-29T23:06:18.287088+0000","last_clean":"2023-09-29T23:06:18.287088+0000","last_became_active":"2023-09-29T23:06:18.279830+0000","last_became_peered":"2023-09-29T23:06:18.279830+0000","last_unstale":"2023-09-29T23:06:18.287088+0000","last_undegraded":"2023-09-29T23:06:18.287088+0000","last_fullsized":"2023-09-29T23:06:18.287088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280206+0000","last_change":"2023-09-29T23:06:18.270148+0000","last_active":"2023-09-29T23:06:18.280206+0000","last_peered":"2023-09-29T23:06:18.280206+0000","last_clean":"2023-09-29T23:06:18.280206+0000","last_became_active":"2023-09-29T23:06:18.269859+0000","last_became_peered":"2023-09-29T23:06:18.269859+0000","last_unstale":"2023-09-29T23:06:18.280206+0000","last_undegraded":"2023-09-29T23:06:18.280206+0000","last_fullsized":"2023-09-29T23:06:18.280206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277323+0000","last_change":"2023-09-29T23:06:18.272855+0000","last_active":"2023-09-29T23:06:18.277323+0000","last_peered":"2023-09-29T23:06:18.277323+0000","last_clean":"2023-09-29T23:06:18.277323+0000","last_became_active":"2023-09-29T23:06:18.272591+0000","last_became_peered":"2023-09-29T23:06:18.272591+0000","last_unstale":"2023-09-29T23:06:18.277323+0000","last_undegraded":"2023-09-29T23:06:18.277323+0000","last_fullsized":"2023-09-29T23:06:18.277323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:35.444407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285404+0000","last_change":"2023-09-29T23:06:18.283392+0000","last_active":"2023-09-29T23:06:18.285404+0000","last_peered":"2023-09-29T23:06:18.285404+0000","last_clean":"2023-09-29T23:06:18.285404+0000","last_became_active":"2023-09-29T23:06:18.282197+0000","last_became_peered":"2023-09-29T23:06:18.282197+0000","last_unstale":"2023-09-29T23:06:18.285404+0000","last_undegraded":"2023-09-29T23:06:18.285404+0000","last_fullsized":"2023-09-29T23:06:18.285404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:33.387327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269370+0000","last_change":"2023-09-29T23:06:18.268361+0000","last_active":"2023-09-29T23:06:18.269370+0000","last_peered":"2023-09-29T23:06:18.269370+0000","last_clean":"2023-09-29T23:06:18.269370+0000","last_became_active":"2023-09-29T23:06:18.268085+0000","last_became_peered":"2023-09-29T23:06:18.268085+0000","last_unstale":"2023-09-29T23:06:18.269370+0000","last_undegraded":"2023-09-29T23:06:18.269370+0000","last_fullsized":"2023-09-29T23:06:18.269370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283344+0000","last_change":"2023-09-29T23:06:18.282177+0000","last_active":"2023-09-29T23:06:18.283344+0000","last_peered":"2023-09-29T23:06:18.283344+0000","last_clean":"2023-09-29T23:06:18.283344+0000","last_became_active":"2023-09-29T23:06:18.281841+0000","last_became_peered":"2023-09-29T23:06:18.281841+0000","last_unstale":"2023-09-29T23:06:18.283344+0000","last_undegraded":"2023-09-29T23:06:18.283344+0000","last_fullsized":"2023-09-29T23:06:18.283344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273852+0000","last_change":"2023-09-29T23:06:18.272693+0000","last_active":"2023-09-29T23:06:18.273852+0000","last_peered":"2023-09-29T23:06:18.273852+0000","last_clean":"2023-09-29T23:06:18.273852+0000","last_became_active":"2023-09-29T23:06:18.272433+0000","last_became_peered":"2023-09-29T23:06:18.272433+0000","last_unstale":"2023-09-29T23:06:18.273852+0000","last_undegraded":"2023-09-29T23:06:18.273852+0000","last_fullsized":"2023-09-29T23:06:18.273852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280760+0000","last_change":"2023-09-29T23:06:18.275372+0000","last_active":"2023-09-29T23:06:18.280760+0000","last_peered":"2023-09-29T23:06:18.280760+0000","last_clean":"2023-09-29T23:06:18.280760+0000","last_became_active":"2023-09-29T23:06:18.274861+0000","last_became_peered":"2023-09-29T23:06:18.274861+0000","last_unstale":"2023-09-29T23:06:18.280760+0000","last_undegraded":"2023-09-29T23:06:18.280760+0000","last_fullsized":"2023-09-29T23:06:18.280760+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290310+0000","last_change":"2023-09-29T23:06:18.287098+0000","last_active":"2023-09-29T23:06:18.290310+0000","last_peered":"2023-09-29T23:06:18.290310+0000","last_clean":"2023-09-29T23:06:18.290310+0000","last_became_active":"2023-09-29T23:06:18.286847+0000","last_became_peered":"2023-09-29T23:06:18.286847+0000","last_unstale":"2023-09-29T23:06:18.290310+0000","last_undegraded":"2023-09-29T23:06:18.290310+0000","last_fullsized":"2023-09-29T23:06:18.290310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277206+0000","last_change":"2023-09-29T23:06:18.273575+0000","last_active":"2023-09-29T23:06:18.277206+0000","last_peered":"2023-09-29T23:06:18.277206+0000","last_clean":"2023-09-29T23:06:18.277206+0000","last_became_active":"2023-09-29T23:06:18.273327+0000","last_became_peered":"2023-09-29T23:06:18.273327+0000","last_unstale":"2023-09-29T23:06:18.277206+0000","last_undegraded":"2023-09-29T23:06:18.277206+0000","last_fullsized":"2023-09-29T23:06:18.277206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:29:05.706848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280438+0000","last_change":"2023-09-29T23:06:18.271980+0000","last_active":"2023-09-29T23:06:18.280438+0000","last_peered":"2023-09-29T23:06:18.280438+0000","last_clean":"2023-09-29T23:06:18.280438+0000","last_became_active":"2023-09-29T23:06:18.271725+0000","last_became_peered":"2023-09-29T23:06:18.271725+0000","last_unstale":"2023-09-29T23:06:18.280438+0000","last_undegraded":"2023-09-29T23:06:18.280438+0000","last_fullsized":"2023-09-29T23:06:18.280438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283120+0000","last_change":"2023-09-29T23:06:18.281173+0000","last_active":"2023-09-29T23:06:18.283120+0000","last_peered":"2023-09-29T23:06:18.283120+0000","last_clean":"2023-09-29T23:06:18.283120+0000","last_became_active":"2023-09-29T23:06:18.280834+0000","last_became_peered":"2023-09-29T23:06:18.280834+0000","last_unstale":"2023-09-29T23:06:18.283120+0000","last_undegraded":"2023-09-29T23:06:18.283120+0000","last_fullsized":"2023-09-29T23:06:18.283120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:02:54.551488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280253+0000","last_change":"2023-09-29T23:06:18.270925+0000","last_active":"2023-09-29T23:06:18.280253+0000","last_peered":"2023-09-29T23:06:18.280253+0000","last_clean":"2023-09-29T23:06:18.280253+0000","last_became_active":"2023-09-29T23:06:18.269605+0000","last_became_peered":"2023-09-29T23:06:18.269605+0000","last_unstale":"2023-09-29T23:06:18.280253+0000","last_undegraded":"2023-09-29T23:06:18.280253+0000","last_fullsized":"2023-09-29T23:06:18.280253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287559+0000","last_change":"2023-09-29T23:06:18.284188+0000","last_active":"2023-09-29T23:06:18.287559+0000","last_peered":"2023-09-29T23:06:18.287559+0000","last_clean":"2023-09-29T23:06:18.287559+0000","last_became_active":"2023-09-29T23:06:18.283875+0000","last_became_peered":"2023-09-29T23:06:18.283875+0000","last_unstale":"2023-09-29T23:06:18.287559+0000","last_undegraded":"2023-09-29T23:06:18.287559+0000","last_fullsized":"2023-09-29T23:06:18.287559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283044+0000","last_change":"2023-09-29T23:06:18.280077+0000","last_active":"2023-09-29T23:06:18.283044+0000","last_peered":"2023-09-29T23:06:18.283044+0000","last_clean":"2023-09-29T23:06:18.283044+0000","last_became_active":"2023-09-29T23:06:18.279865+0000","last_became_peered":"2023-09-29T23:06:18.279865+0000","last_unstale":"2023-09-29T23:06:18.283044+0000","last_undegraded":"2023-09-29T23:06:18.283044+0000","last_fullsized":"2023-09-29T23:06:18.283044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:48.644161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280240+0000","last_change":"2023-09-29T23:06:18.270238+0000","last_active":"2023-09-29T23:06:18.280240+0000","last_peered":"2023-09-29T23:06:18.280240+0000","last_clean":"2023-09-29T23:06:18.280240+0000","last_became_active":"2023-09-29T23:06:18.269996+0000","last_became_peered":"2023-09-29T23:06:18.269996+0000","last_unstale":"2023-09-29T23:06:18.280240+0000","last_undegraded":"2023-09-29T23:06:18.280240+0000","last_fullsized":"2023-09-29T23:06:18.280240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287050+0000","last_change":"2023-09-29T23:06:18.279595+0000","last_active":"2023-09-29T23:06:18.287050+0000","last_peered":"2023-09-29T23:06:18.287050+0000","last_clean":"2023-09-29T23:06:18.287050+0000","last_became_active":"2023-09-29T23:06:18.278827+0000","last_became_peered":"2023-09-29T23:06:18.278827+0000","last_unstale":"2023-09-29T23:06:18.287050+0000","last_undegraded":"2023-09-29T23:06:18.287050+0000","last_fullsized":"2023-09-29T23:06:18.287050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288146+0000","last_change":"2023-09-29T23:06:18.287081+0000","last_active":"2023-09-29T23:06:18.288146+0000","last_peered":"2023-09-29T23:06:18.288146+0000","last_clean":"2023-09-29T23:06:18.288146+0000","last_became_active":"2023-09-29T23:06:18.286853+0000","last_became_peered":"2023-09-29T23:06:18.286853+0000","last_unstale":"2023-09-29T23:06:18.288146+0000","last_undegraded":"2023-09-29T23:06:18.288146+0000","last_fullsized":"2023-09-29T23:06:18.288146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:00.791418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280662+0000","last_change":"2023-09-29T23:06:17.886342+0000","last_active":"2023-09-29T23:06:18.280662+0000","last_peered":"2023-09-29T23:06:18.280662+0000","last_clean":"2023-09-29T23:06:18.280662+0000","last_became_active":"2023-09-29T23:06:17.886089+0000","last_became_peered":"2023-09-29T23:06:17.886089+0000","last_unstale":"2023-09-29T23:06:18.280662+0000","last_undegraded":"2023-09-29T23:06:18.280662+0000","last_fullsized":"2023-09-29T23:06:18.280662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:04.152750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288614+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288301+0000","last_became_peered":"2023-09-29T23:06:18.288301+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261493+0000","last_change":"2023-09-29T23:06:17.891522+0000","last_active":"2023-09-29T23:06:18.261493+0000","last_peered":"2023-09-29T23:06:18.261493+0000","last_clean":"2023-09-29T23:06:18.261493+0000","last_became_active":"2023-09-29T23:06:17.891232+0000","last_became_peered":"2023-09-29T23:06:17.891232+0000","last_unstale":"2023-09-29T23:06:18.261493+0000","last_undegraded":"2023-09-29T23:06:18.261493+0000","last_fullsized":"2023-09-29T23:06:18.261493+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287070+0000","last_change":"2023-09-29T23:06:18.279774+0000","last_active":"2023-09-29T23:06:18.287070+0000","last_peered":"2023-09-29T23:06:18.287070+0000","last_clean":"2023-09-29T23:06:18.287070+0000","last_became_active":"2023-09-29T23:06:18.279014+0000","last_became_peered":"2023-09-29T23:06:18.279014+0000","last_unstale":"2023-09-29T23:06:18.287070+0000","last_undegraded":"2023-09-29T23:06:18.287070+0000","last_fullsized":"2023-09-29T23:06:18.287070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273886+0000","last_change":"2023-09-29T23:06:17.894216+0000","last_active":"2023-09-29T23:06:18.273886+0000","last_peered":"2023-09-29T23:06:18.273886+0000","last_clean":"2023-09-29T23:06:18.273886+0000","last_became_active":"2023-09-29T23:06:17.893923+0000","last_became_peered":"2023-09-29T23:06:17.893923+0000","last_unstale":"2023-09-29T23:06:18.273886+0000","last_undegraded":"2023-09-29T23:06:18.273886+0000","last_fullsized":"2023-09-29T23:06:18.273886+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282454+0000","last_change":"2023-09-29T23:06:18.276320+0000","last_active":"2023-09-29T23:06:18.282454+0000","last_peered":"2023-09-29T23:06:18.282454+0000","last_clean":"2023-09-29T23:06:18.282454+0000","last_became_active":"2023-09-29T23:06:18.276049+0000","last_became_peered":"2023-09-29T23:06:18.276049+0000","last_unstale":"2023-09-29T23:06:18.282454+0000","last_undegraded":"2023-09-29T23:06:18.282454+0000","last_fullsized":"2023-09-29T23:06:18.282454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:32.688280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261797+0000","last_change":"2023-09-29T23:06:17.883845+0000","last_active":"2023-09-29T23:06:18.261797+0000","last_peered":"2023-09-29T23:06:18.261797+0000","last_clean":"2023-09-29T23:06:18.261797+0000","last_became_active":"2023-09-29T23:06:17.883559+0000","last_became_peered":"2023-09-29T23:06:17.883559+0000","last_unstale":"2023-09-29T23:06:18.261797+0000","last_undegraded":"2023-09-29T23:06:18.261797+0000","last_fullsized":"2023-09-29T23:06:18.261797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286481+0000","last_change":"2023-09-29T23:06:18.284038+0000","last_active":"2023-09-29T23:06:18.286481+0000","last_peered":"2023-09-29T23:06:18.286481+0000","last_clean":"2023-09-29T23:06:18.286481+0000","last_became_active":"2023-09-29T23:06:18.283764+0000","last_became_peered":"2023-09-29T23:06:18.283764+0000","last_unstale":"2023-09-29T23:06:18.286481+0000","last_undegraded":"2023-09-29T23:06:18.286481+0000","last_fullsized":"2023-09-29T23:06:18.286481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:05:42.345774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261094+0000","last_change":"2023-09-29T23:06:17.886414+0000","last_active":"2023-09-29T23:06:18.261094+0000","last_peered":"2023-09-29T23:06:18.261094+0000","last_clean":"2023-09-29T23:06:18.261094+0000","last_became_active":"2023-09-29T23:06:17.886124+0000","last_became_peered":"2023-09-29T23:06:17.886124+0000","last_unstale":"2023-09-29T23:06:18.261094+0000","last_undegraded":"2023-09-29T23:06:18.261094+0000","last_fullsized":"2023-09-29T23:06:18.261094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:32.385528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288784+0000","last_change":"2023-09-29T23:06:18.286164+0000","last_active":"2023-09-29T23:06:18.288784+0000","last_peered":"2023-09-29T23:06:18.288784+0000","last_clean":"2023-09-29T23:06:18.288784+0000","last_became_active":"2023-09-29T23:06:18.285518+0000","last_became_peered":"2023-09-29T23:06:18.285518+0000","last_unstale":"2023-09-29T23:06:18.288784+0000","last_undegraded":"2023-09-29T23:06:18.288784+0000","last_fullsized":"2023-09-29T23:06:18.288784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:54:18.859428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286862+0000","last_change":"2023-09-29T23:06:17.889196+0000","last_active":"2023-09-29T23:06:18.286862+0000","last_peered":"2023-09-29T23:06:18.286862+0000","last_clean":"2023-09-29T23:06:18.286862+0000","last_became_active":"2023-09-29T23:06:17.888932+0000","last_became_peered":"2023-09-29T23:06:17.888932+0000","last_unstale":"2023-09-29T23:06:18.286862+0000","last_undegraded":"2023-09-29T23:06:18.286862+0000","last_fullsized":"2023-09-29T23:06:18.286862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280276+0000","last_change":"2023-09-29T23:06:18.271028+0000","last_active":"2023-09-29T23:06:18.280276+0000","last_peered":"2023-09-29T23:06:18.280276+0000","last_clean":"2023-09-29T23:06:18.280276+0000","last_became_active":"2023-09-29T23:06:18.270012+0000","last_became_peered":"2023-09-29T23:06:18.270012+0000","last_unstale":"2023-09-29T23:06:18.280276+0000","last_undegraded":"2023-09-29T23:06:18.280276+0000","last_fullsized":"2023-09-29T23:06:18.280276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.293088+0000","last_change":"2023-09-29T23:06:17.889224+0000","last_active":"2023-09-29T23:06:18.293088+0000","last_peered":"2023-09-29T23:06:18.293088+0000","last_clean":"2023-09-29T23:06:18.293088+0000","last_became_active":"2023-09-29T23:06:17.888941+0000","last_became_peered":"2023-09-29T23:06:17.888941+0000","last_unstale":"2023-09-29T23:06:18.293088+0000","last_undegraded":"2023-09-29T23:06:18.293088+0000","last_fullsized":"2023-09-29T23:06:18.293088+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:06:09.687876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274555+0000","last_change":"2023-09-29T23:06:18.270435+0000","last_active":"2023-09-29T23:06:18.274555+0000","last_peered":"2023-09-29T23:06:18.274555+0000","last_clean":"2023-09-29T23:06:18.274555+0000","last_became_active":"2023-09-29T23:06:18.268172+0000","last_became_peered":"2023-09-29T23:06:18.268172+0000","last_unstale":"2023-09-29T23:06:18.274555+0000","last_undegraded":"2023-09-29T23:06:18.274555+0000","last_fullsized":"2023-09-29T23:06:18.274555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:45.768259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289061+0000","last_change":"2023-09-29T23:06:17.893695+0000","last_active":"2023-09-29T23:06:18.289061+0000","last_peered":"2023-09-29T23:06:18.289061+0000","last_clean":"2023-09-29T23:06:18.289061+0000","last_became_active":"2023-09-29T23:06:17.893381+0000","last_became_peered":"2023-09-29T23:06:17.893381+0000","last_unstale":"2023-09-29T23:06:18.289061+0000","last_undegraded":"2023-09-29T23:06:18.289061+0000","last_fullsized":"2023-09-29T23:06:18.289061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292208+0000","last_change":"2023-09-29T23:06:18.287485+0000","last_active":"2023-09-29T23:06:18.292208+0000","last_peered":"2023-09-29T23:06:18.292208+0000","last_clean":"2023-09-29T23:06:18.292208+0000","last_became_active":"2023-09-29T23:06:18.285220+0000","last_became_peered":"2023-09-29T23:06:18.285220+0000","last_unstale":"2023-09-29T23:06:18.292208+0000","last_undegraded":"2023-09-29T23:06:18.292208+0000","last_fullsized":"2023-09-29T23:06:18.292208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260745+0000","last_change":"2023-09-29T23:06:17.888841+0000","last_active":"2023-09-29T23:06:18.260745+0000","last_peered":"2023-09-29T23:06:18.260745+0000","last_clean":"2023-09-29T23:06:18.260745+0000","last_became_active":"2023-09-29T23:06:17.888515+0000","last_became_peered":"2023-09-29T23:06:17.888515+0000","last_unstale":"2023-09-29T23:06:18.260745+0000","last_undegraded":"2023-09-29T23:06:18.260745+0000","last_fullsized":"2023-09-29T23:06:18.260745+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287868+0000","last_change":"2023-09-29T23:06:18.285208+0000","last_active":"2023-09-29T23:06:18.287868+0000","last_peered":"2023-09-29T23:06:18.287868+0000","last_clean":"2023-09-29T23:06:18.287868+0000","last_became_active":"2023-09-29T23:06:18.284879+0000","last_became_peered":"2023-09-29T23:06:18.284879+0000","last_unstale":"2023-09-29T23:06:18.287868+0000","last_undegraded":"2023-09-29T23:06:18.287868+0000","last_fullsized":"2023-09-29T23:06:18.287868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260580+0000","last_change":"2023-09-29T23:06:17.890315+0000","last_active":"2023-09-29T23:06:18.260580+0000","last_peered":"2023-09-29T23:06:18.260580+0000","last_clean":"2023-09-29T23:06:18.260580+0000","last_became_active":"2023-09-29T23:06:17.890017+0000","last_became_peered":"2023-09-29T23:06:17.890017+0000","last_unstale":"2023-09-29T23:06:18.260580+0000","last_undegraded":"2023-09-29T23:06:18.260580+0000","last_fullsized":"2023-09-29T23:06:18.260580+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:55:35.860921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288529+0000","last_change":"2023-09-29T23:06:18.285771+0000","last_active":"2023-09-29T23:06:18.288529+0000","last_peered":"2023-09-29T23:06:18.288529+0000","last_clean":"2023-09-29T23:06:18.288529+0000","last_became_active":"2023-09-29T23:06:18.285493+0000","last_became_peered":"2023-09-29T23:06:18.285493+0000","last_unstale":"2023-09-29T23:06:18.288529+0000","last_undegraded":"2023-09-29T23:06:18.288529+0000","last_fullsized":"2023-09-29T23:06:18.288529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282000+0000","last_change":"2023-09-29T23:06:17.891835+0000","last_active":"2023-09-29T23:06:18.282000+0000","last_peered":"2023-09-29T23:06:18.282000+0000","last_clean":"2023-09-29T23:06:18.282000+0000","last_became_active":"2023-09-29T23:06:17.891563+0000","last_became_peered":"2023-09-29T23:06:17.891563+0000","last_unstale":"2023-09-29T23:06:18.282000+0000","last_undegraded":"2023-09-29T23:06:18.282000+0000","last_fullsized":"2023-09-29T23:06:18.282000+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:22.597416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292423+0000","last_change":"2023-09-29T23:06:18.289343+0000","last_active":"2023-09-29T23:06:18.292423+0000","last_peered":"2023-09-29T23:06:18.292423+0000","last_clean":"2023-09-29T23:06:18.292423+0000","last_became_active":"2023-09-29T23:06:18.289072+0000","last_became_peered":"2023-09-29T23:06:18.289072+0000","last_unstale":"2023-09-29T23:06:18.292423+0000","last_undegraded":"2023-09-29T23:06:18.292423+0000","last_fullsized":"2023-09-29T23:06:18.292423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:27:12.781947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282222+0000","last_change":"2023-09-29T23:06:17.891844+0000","last_active":"2023-09-29T23:06:18.282222+0000","last_peered":"2023-09-29T23:06:18.282222+0000","last_clean":"2023-09-29T23:06:18.282222+0000","last_became_active":"2023-09-29T23:06:17.891582+0000","last_became_peered":"2023-09-29T23:06:17.891582+0000","last_unstale":"2023-09-29T23:06:18.282222+0000","last_undegraded":"2023-09-29T23:06:18.282222+0000","last_fullsized":"2023-09-29T23:06:18.282222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:59.037943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287898+0000","last_change":"2023-09-29T23:06:18.285944+0000","last_active":"2023-09-29T23:06:18.287898+0000","last_peered":"2023-09-29T23:06:18.287898+0000","last_clean":"2023-09-29T23:06:18.287898+0000","last_became_active":"2023-09-29T23:06:18.285661+0000","last_became_peered":"2023-09-29T23:06:18.285661+0000","last_unstale":"2023-09-29T23:06:18.287898+0000","last_undegraded":"2023-09-29T23:06:18.287898+0000","last_fullsized":"2023-09-29T23:06:18.287898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:20:25.540032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.275819+0000","last_change":"2023-09-29T23:06:17.889165+0000","last_active":"2023-09-29T23:06:18.275819+0000","last_peered":"2023-09-29T23:06:18.275819+0000","last_clean":"2023-09-29T23:06:18.275819+0000","last_became_active":"2023-09-29T23:06:17.888842+0000","last_became_peered":"2023-09-29T23:06:17.888842+0000","last_unstale":"2023-09-29T23:06:18.275819+0000","last_undegraded":"2023-09-29T23:06:18.275819+0000","last_fullsized":"2023-09-29T23:06:18.275819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:35.202255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287772+0000","last_change":"2023-09-29T23:06:18.277402+0000","last_active":"2023-09-29T23:06:18.287772+0000","last_peered":"2023-09-29T23:06:18.287772+0000","last_clean":"2023-09-29T23:06:18.287772+0000","last_became_active":"2023-09-29T23:06:18.277159+0000","last_became_peered":"2023-09-29T23:06:18.277159+0000","last_unstale":"2023-09-29T23:06:18.287772+0000","last_undegraded":"2023-09-29T23:06:18.287772+0000","last_fullsized":"2023-09-29T23:06:18.287772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273008+0000","last_change":"2023-09-29T23:06:17.890237+0000","last_active":"2023-09-29T23:06:18.273008+0000","last_peered":"2023-09-29T23:06:18.273008+0000","last_clean":"2023-09-29T23:06:18.273008+0000","last_became_active":"2023-09-29T23:06:17.889971+0000","last_became_peered":"2023-09-29T23:06:17.889971+0000","last_unstale":"2023-09-29T23:06:18.273008+0000","last_undegraded":"2023-09-29T23:06:18.273008+0000","last_fullsized":"2023-09-29T23:06:18.273008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:57.168709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282466+0000","last_change":"2023-09-29T23:06:18.276262+0000","last_active":"2023-09-29T23:06:18.282466+0000","last_peered":"2023-09-29T23:06:18.282466+0000","last_clean":"2023-09-29T23:06:18.282466+0000","last_became_active":"2023-09-29T23:06:18.276040+0000","last_became_peered":"2023-09-29T23:06:18.276040+0000","last_unstale":"2023-09-29T23:06:18.282466+0000","last_undegraded":"2023-09-29T23:06:18.282466+0000","last_fullsized":"2023-09-29T23:06:18.282466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260885+0000","last_change":"2023-09-29T23:06:17.888828+0000","last_active":"2023-09-29T23:06:18.260885+0000","last_peered":"2023-09-29T23:06:18.260885+0000","last_clean":"2023-09-29T23:06:18.260885+0000","last_became_active":"2023-09-29T23:06:17.888525+0000","last_became_peered":"2023-09-29T23:06:17.888525+0000","last_unstale":"2023-09-29T23:06:18.260885+0000","last_undegraded":"2023-09-29T23:06:18.260885+0000","last_fullsized":"2023-09-29T23:06:18.260885+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:18:04.884961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285419+0000","last_change":"2023-09-29T23:06:18.283530+0000","last_active":"2023-09-29T23:06:18.285419+0000","last_peered":"2023-09-29T23:06:18.285419+0000","last_clean":"2023-09-29T23:06:18.285419+0000","last_became_active":"2023-09-29T23:06:18.281248+0000","last_became_peered":"2023-09-29T23:06:18.281248+0000","last_unstale":"2023-09-29T23:06:18.285419+0000","last_undegraded":"2023-09-29T23:06:18.285419+0000","last_fullsized":"2023-09-29T23:06:18.285419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:23.599215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289625+0000","last_change":"2023-09-29T23:06:17.894026+0000","last_active":"2023-09-29T23:06:18.289625+0000","last_peered":"2023-09-29T23:06:18.289625+0000","last_clean":"2023-09-29T23:06:18.289625+0000","last_became_active":"2023-09-29T23:06:17.893757+0000","last_became_peered":"2023-09-29T23:06:17.893757+0000","last_unstale":"2023-09-29T23:06:18.289625+0000","last_undegraded":"2023-09-29T23:06:18.289625+0000","last_fullsized":"2023-09-29T23:06:18.289625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286633+0000","last_change":"2023-09-29T23:06:18.278135+0000","last_active":"2023-09-29T23:06:18.286633+0000","last_peered":"2023-09-29T23:06:18.286633+0000","last_clean":"2023-09-29T23:06:18.286633+0000","last_became_active":"2023-09-29T23:06:18.277865+0000","last_became_peered":"2023-09-29T23:06:18.277865+0000","last_unstale":"2023-09-29T23:06:18.286633+0000","last_undegraded":"2023-09-29T23:06:18.286633+0000","last_fullsized":"2023-09-29T23:06:18.286633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260719+0000","last_change":"2023-09-29T23:06:17.891680+0000","last_active":"2023-09-29T23:06:18.260719+0000","last_peered":"2023-09-29T23:06:18.260719+0000","last_clean":"2023-09-29T23:06:18.260719+0000","last_became_active":"2023-09-29T23:06:17.891413+0000","last_became_peered":"2023-09-29T23:06:17.891413+0000","last_unstale":"2023-09-29T23:06:18.260719+0000","last_undegraded":"2023-09-29T23:06:18.260719+0000","last_fullsized":"2023-09-29T23:06:18.260719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:19.905553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269364+0000","last_change":"2023-09-29T23:06:18.268288+0000","last_active":"2023-09-29T23:06:18.269364+0000","last_peered":"2023-09-29T23:06:18.269364+0000","last_clean":"2023-09-29T23:06:18.269364+0000","last_became_active":"2023-09-29T23:06:18.268079+0000","last_became_peered":"2023-09-29T23:06:18.268079+0000","last_unstale":"2023-09-29T23:06:18.269364+0000","last_undegraded":"2023-09-29T23:06:18.269364+0000","last_fullsized":"2023-09-29T23:06:18.269364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274260+0000","last_change":"2023-09-29T23:06:17.892193+0000","last_active":"2023-09-29T23:06:18.274260+0000","last_peered":"2023-09-29T23:06:18.274260+0000","last_clean":"2023-09-29T23:06:18.274260+0000","last_became_active":"2023-09-29T23:06:17.891898+0000","last_became_peered":"2023-09-29T23:06:17.891898+0000","last_unstale":"2023-09-29T23:06:18.274260+0000","last_undegraded":"2023-09-29T23:06:18.274260+0000","last_fullsized":"2023-09-29T23:06:18.274260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.276612+0000","last_change":"2023-09-29T23:06:18.271513+0000","last_active":"2023-09-29T23:06:18.276612+0000","last_peered":"2023-09-29T23:06:18.276612+0000","last_clean":"2023-09-29T23:06:18.276612+0000","last_became_active":"2023-09-29T23:06:18.271257+0000","last_became_peered":"2023-09-29T23:06:18.271257+0000","last_unstale":"2023-09-29T23:06:18.276612+0000","last_undegraded":"2023-09-29T23:06:18.276612+0000","last_fullsized":"2023-09-29T23:06:18.276612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:23.301482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273504+0000","last_change":"2023-09-29T23:06:17.893616+0000","last_active":"2023-09-29T23:06:18.273504+0000","last_peered":"2023-09-29T23:06:18.273504+0000","last_clean":"2023-09-29T23:06:18.273504+0000","last_became_active":"2023-09-29T23:06:17.893285+0000","last_became_peered":"2023-09-29T23:06:17.893285+0000","last_unstale":"2023-09-29T23:06:18.273504+0000","last_undegraded":"2023-09-29T23:06:18.273504+0000","last_fullsized":"2023-09-29T23:06:18.273504+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283690+0000","last_change":"2023-09-29T23:06:18.277473+0000","last_active":"2023-09-29T23:06:18.283690+0000","last_peered":"2023-09-29T23:06:18.283690+0000","last_clean":"2023-09-29T23:06:18.283690+0000","last_became_active":"2023-09-29T23:06:18.276953+0000","last_became_peered":"2023-09-29T23:06:18.276953+0000","last_unstale":"2023-09-29T23:06:18.283690+0000","last_undegraded":"2023-09-29T23:06:18.283690+0000","last_fullsized":"2023-09-29T23:06:18.283690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264133+0000","last_change":"2023-09-29T23:06:13.639723+0000","last_active":"2023-09-29T23:06:18.264133+0000","last_peered":"2023-09-29T23:06:18.264133+0000","last_clean":"2023-09-29T23:06:18.264133+0000","last_became_active":"2023-09-29T23:06:13.639406+0000","last_became_peered":"2023-09-29T23:06:13.639406+0000","last_unstale":"2023-09-29T23:06:18.264133+0000","last_undegraded":"2023-09-29T23:06:18.264133+0000","last_fullsized":"2023-09-29T23:06:18.264133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:07:23.007465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261412+0000","last_change":"2023-09-29T23:06:17.894258+0000","last_active":"2023-09-29T23:06:18.261412+0000","last_peered":"2023-09-29T23:06:18.261412+0000","last_clean":"2023-09-29T23:06:18.261412+0000","last_became_active":"2023-09-29T23:06:17.893935+0000","last_became_peered":"2023-09-29T23:06:17.893935+0000","last_unstale":"2023-09-29T23:06:18.261412+0000","last_undegraded":"2023-09-29T23:06:18.261412+0000","last_fullsized":"2023-09-29T23:06:18.261412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:30:41.674282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280296+0000","last_change":"2023-09-29T23:06:18.271186+0000","last_active":"2023-09-29T23:06:18.280296+0000","last_peered":"2023-09-29T23:06:18.280296+0000","last_clean":"2023-09-29T23:06:18.280296+0000","last_became_active":"2023-09-29T23:06:18.270718+0000","last_became_peered":"2023-09-29T23:06:18.270718+0000","last_unstale":"2023-09-29T23:06:18.280296+0000","last_undegraded":"2023-09-29T23:06:18.280296+0000","last_fullsized":"2023-09-29T23:06:18.280296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:58.170509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260686+0000","last_change":"2023-09-29T23:06:13.380282+0000","last_active":"2023-09-29T23:06:18.260686+0000","last_peered":"2023-09-29T23:06:18.260686+0000","last_clean":"2023-09-29T23:06:18.260686+0000","last_became_active":"2023-09-29T23:06:13.379948+0000","last_became_peered":"2023-09-29T23:06:13.379948+0000","last_unstale":"2023-09-29T23:06:18.260686+0000","last_undegraded":"2023-09-29T23:06:18.260686+0000","last_fullsized":"2023-09-29T23:06:18.260686+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:02.759420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261279+0000","last_change":"2023-09-29T23:06:17.884199+0000","last_active":"2023-09-29T23:06:18.261279+0000","last_peered":"2023-09-29T23:06:18.261279+0000","last_clean":"2023-09-29T23:06:18.261279+0000","last_became_active":"2023-09-29T23:06:17.883932+0000","last_became_peered":"2023-09-29T23:06:17.883932+0000","last_unstale":"2023-09-29T23:06:18.261279+0000","last_undegraded":"2023-09-29T23:06:18.261279+0000","last_fullsized":"2023-09-29T23:06:18.261279+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:39.182159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283703+0000","last_change":"2023-09-29T23:06:18.277498+0000","last_active":"2023-09-29T23:06:18.283703+0000","last_peered":"2023-09-29T23:06:18.283703+0000","last_clean":"2023-09-29T23:06:18.283703+0000","last_became_active":"2023-09-29T23:06:18.276885+0000","last_became_peered":"2023-09-29T23:06:18.276885+0000","last_unstale":"2023-09-29T23:06:18.283703+0000","last_undegraded":"2023-09-29T23:06:18.283703+0000","last_fullsized":"2023-09-29T23:06:18.283703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.259675+0000","last_change":"2023-09-29T23:06:13.639753+0000","last_active":"2023-09-29T23:06:18.259675+0000","last_peered":"2023-09-29T23:06:18.259675+0000","last_clean":"2023-09-29T23:06:18.259675+0000","last_became_active":"2023-09-29T23:06:13.639436+0000","last_became_peered":"2023-09-29T23:06:13.639436+0000","last_unstale":"2023-09-29T23:06:18.259675+0000","last_undegraded":"2023-09-29T23:06:18.259675+0000","last_fullsized":"2023-09-29T23:06:18.259675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:09:57.060849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261100+0000","last_change":"2023-09-29T23:06:17.885616+0000","last_active":"2023-09-29T23:06:18.261100+0000","last_peered":"2023-09-29T23:06:18.261100+0000","last_clean":"2023-09-29T23:06:18.261100+0000","last_became_active":"2023-09-29T23:06:17.885342+0000","last_became_peered":"2023-09-29T23:06:17.885342+0000","last_unstale":"2023-09-29T23:06:18.261100+0000","last_undegraded":"2023-09-29T23:06:18.261100+0000","last_fullsized":"2023-09-29T23:06:18.261100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280779+0000","last_change":"2023-09-29T23:06:18.277223+0000","last_active":"2023-09-29T23:06:18.280779+0000","last_peered":"2023-09-29T23:06:18.280779+0000","last_clean":"2023-09-29T23:06:18.280779+0000","last_became_active":"2023-09-29T23:06:18.276548+0000","last_became_peered":"2023-09-29T23:06:18.276548+0000","last_unstale":"2023-09-29T23:06:18.280779+0000","last_undegraded":"2023-09-29T23:06:18.280779+0000","last_fullsized":"2023-09-29T23:06:18.280779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261249+0000","last_change":"2023-09-29T23:06:13.380967+0000","last_active":"2023-09-29T23:06:18.261249+0000","last_peered":"2023-09-29T23:06:18.261249+0000","last_clean":"2023-09-29T23:06:18.261249+0000","last_became_active":"2023-09-29T23:06:13.379890+0000","last_became_peered":"2023-09-29T23:06:13.379890+0000","last_unstale":"2023-09-29T23:06:18.261249+0000","last_undegraded":"2023-09-29T23:06:18.261249+0000","last_fullsized":"2023-09-29T23:06:18.261249+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:46.738824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260986+0000","last_change":"2023-09-29T23:06:09.717533+0000","last_active":"2023-09-29T23:06:18.260986+0000","last_peered":"2023-09-29T23:06:18.260986+0000","last_clean":"2023-09-29T23:06:18.260986+0000","last_became_active":"2023-09-29T23:06:09.716536+0000","last_became_peered":"2023-09-29T23:06:09.716536+0000","last_unstale":"2023-09-29T23:06:18.260986+0000","last_undegraded":"2023-09-29T23:06:18.260986+0000","last_fullsized":"2023-09-29T23:06:18.260986+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_clean_scrub_stamp":"2023-09-29T23:06:08.290533+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:44:31.128783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274816+0000","last_change":"2023-09-29T23:06:17.892420+0000","last_active":"2023-09-29T23:06:18.274816+0000","last_peered":"2023-09-29T23:06:18.274816+0000","last_clean":"2023-09-29T23:06:18.274816+0000","last_became_active":"2023-09-29T23:06:17.892121+0000","last_became_peered":"2023-09-29T23:06:17.892121+0000","last_unstale":"2023-09-29T23:06:18.274816+0000","last_undegraded":"2023-09-29T23:06:18.274816+0000","last_fullsized":"2023-09-29T23:06:18.274816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283140+0000","last_change":"2023-09-29T23:06:18.281364+0000","last_active":"2023-09-29T23:06:18.283140+0000","last_peered":"2023-09-29T23:06:18.283140+0000","last_clean":"2023-09-29T23:06:18.283140+0000","last_became_active":"2023-09-29T23:06:18.280821+0000","last_became_peered":"2023-09-29T23:06:18.280821+0000","last_unstale":"2023-09-29T23:06:18.283140+0000","last_undegraded":"2023-09-29T23:06:18.283140+0000","last_fullsized":"2023-09-29T23:06:18.283140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260488+0000","last_change":"2023-09-29T23:06:13.380209+0000","last_active":"2023-09-29T23:06:18.260488+0000","last_peered":"2023-09-29T23:06:18.260488+0000","last_clean":"2023-09-29T23:06:18.260488+0000","last_became_active":"2023-09-29T23:06:13.379845+0000","last_became_peered":"2023-09-29T23:06:13.379845+0000","last_unstale":"2023-09-29T23:06:18.260488+0000","last_undegraded":"2023-09-29T23:06:18.260488+0000","last_fullsized":"2023-09-29T23:06:18.260488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261391+0000","last_change":"2023-09-29T23:06:17.891971+0000","last_active":"2023-09-29T23:06:18.261391+0000","last_peered":"2023-09-29T23:06:18.261391+0000","last_clean":"2023-09-29T23:06:18.261391+0000","last_became_active":"2023-09-29T23:06:17.891695+0000","last_became_peered":"2023-09-29T23:06:17.891695+0000","last_unstale":"2023-09-29T23:06:18.261391+0000","last_undegraded":"2023-09-29T23:06:18.261391+0000","last_fullsized":"2023-09-29T23:06:18.261391+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288649+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288351+0000","last_became_peered":"2023-09-29T23:06:18.288351+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261483+0000","last_change":"2023-09-29T23:06:13.380613+0000","last_active":"2023-09-29T23:06:18.261483+0000","last_peered":"2023-09-29T23:06:18.261483+0000","last_clean":"2023-09-29T23:06:18.261483+0000","last_became_active":"2023-09-29T23:06:13.380347+0000","last_became_peered":"2023-09-29T23:06:13.380347+0000","last_unstale":"2023-09-29T23:06:18.261483+0000","last_undegraded":"2023-09-29T23:06:18.261483+0000","last_fullsized":"2023-09-29T23:06:18.261483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:35.568211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264019+0000","last_change":"2023-09-29T23:06:17.885489+0000","last_active":"2023-09-29T23:06:18.264019+0000","last_peered":"2023-09-29T23:06:18.264019+0000","last_clean":"2023-09-29T23:06:18.264019+0000","last_became_active":"2023-09-29T23:06:17.884966+0000","last_became_peered":"2023-09-29T23:06:17.884966+0000","last_unstale":"2023-09-29T23:06:18.264019+0000","last_undegraded":"2023-09-29T23:06:18.264019+0000","last_fullsized":"2023-09-29T23:06:18.264019+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286897+0000","last_change":"2023-09-29T23:06:18.282836+0000","last_active":"2023-09-29T23:06:18.286897+0000","last_peered":"2023-09-29T23:06:18.286897+0000","last_clean":"2023-09-29T23:06:18.286897+0000","last_became_active":"2023-09-29T23:06:18.282215+0000","last_became_peered":"2023-09-29T23:06:18.282215+0000","last_unstale":"2023-09-29T23:06:18.286897+0000","last_undegraded":"2023-09-29T23:06:18.286897+0000","last_fullsized":"2023-09-29T23:06:18.286897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288992+0000","last_change":"2023-09-29T23:06:17.894025+0000","last_active":"2023-09-29T23:06:18.288992+0000","last_peered":"2023-09-29T23:06:18.288992+0000","last_clean":"2023-09-29T23:06:18.288992+0000","last_became_active":"2023-09-29T23:06:17.893701+0000","last_became_peered":"2023-09-29T23:06:17.893701+0000","last_unstale":"2023-09-29T23:06:18.288992+0000","last_undegraded":"2023-09-29T23:06:18.288992+0000","last_fullsized":"2023-09-29T23:06:18.288992+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.270574+0000","last_change":"2023-09-29T23:06:13.379718+0000","last_active":"2023-09-29T23:06:18.270574+0000","last_peered":"2023-09-29T23:06:18.270574+0000","last_clean":"2023-09-29T23:06:18.270574+0000","last_became_active":"2023-09-29T23:06:13.379450+0000","last_became_peered":"2023-09-29T23:06:13.379450+0000","last_unstale":"2023-09-29T23:06:18.270574+0000","last_undegraded":"2023-09-29T23:06:18.270574+0000","last_fullsized":"2023-09-29T23:06:18.270574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286470+0000","last_change":"2023-09-29T23:06:18.283988+0000","last_active":"2023-09-29T23:06:18.286470+0000","last_peered":"2023-09-29T23:06:18.286470+0000","last_clean":"2023-09-29T23:06:18.286470+0000","last_became_active":"2023-09-29T23:06:18.283448+0000","last_became_peered":"2023-09-29T23:06:18.283448+0000","last_unstale":"2023-09-29T23:06:18.286470+0000","last_undegraded":"2023-09-29T23:06:18.286470+0000","last_fullsized":"2023-09-29T23:06:18.286470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260973+0000","last_change":"2023-09-29T23:06:13.382829+0000","last_active":"2023-09-29T23:06:18.260973+0000","last_peered":"2023-09-29T23:06:18.260973+0000","last_clean":"2023-09-29T23:06:18.260973+0000","last_became_active":"2023-09-29T23:06:13.382590+0000","last_became_peered":"2023-09-29T23:06:13.382590+0000","last_unstale":"2023-09-29T23:06:18.260973+0000","last_undegraded":"2023-09-29T23:06:18.260973+0000","last_fullsized":"2023-09-29T23:06:18.260973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:06.822425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.271596+0000","last_change":"2023-09-29T23:06:17.881636+0000","last_active":"2023-09-29T23:06:18.271596+0000","last_peered":"2023-09-29T23:06:18.271596+0000","last_clean":"2023-09-29T23:06:18.271596+0000","last_became_active":"2023-09-29T23:06:17.881343+0000","last_became_peered":"2023-09-29T23:06:17.881343+0000","last_unstale":"2023-09-29T23:06:18.271596+0000","last_undegraded":"2023-09-29T23:06:18.271596+0000","last_fullsized":"2023-09-29T23:06:18.271596+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274553+0000","last_change":"2023-09-29T23:06:18.268514+0000","last_active":"2023-09-29T23:06:18.274553+0000","last_peered":"2023-09-29T23:06:18.274553+0000","last_clean":"2023-09-29T23:06:18.274553+0000","last_became_active":"2023-09-29T23:06:18.268089+0000","last_became_peered":"2023-09-29T23:06:18.268089+0000","last_unstale":"2023-09-29T23:06:18.274553+0000","last_undegraded":"2023-09-29T23:06:18.274553+0000","last_fullsized":"2023-09-29T23:06:18.274553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:22.930009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288913+0000","last_change":"2023-09-29T23:06:17.892888+0000","last_active":"2023-09-29T23:06:18.288913+0000","last_peered":"2023-09-29T23:06:18.288913+0000","last_clean":"2023-09-29T23:06:18.288913+0000","last_became_active":"2023-09-29T23:06:17.892666+0000","last_became_peered":"2023-09-29T23:06:17.892666+0000","last_unstale":"2023-09-29T23:06:18.288913+0000","last_undegraded":"2023-09-29T23:06:18.288913+0000","last_fullsized":"2023-09-29T23:06:18.288913+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285341+0000","last_change":"2023-09-29T23:06:18.282028+0000","last_active":"2023-09-29T23:06:18.285341+0000","last_peered":"2023-09-29T23:06:18.285341+0000","last_clean":"2023-09-29T23:06:18.285341+0000","last_became_active":"2023-09-29T23:06:18.281546+0000","last_became_peered":"2023-09-29T23:06:18.281546+0000","last_unstale":"2023-09-29T23:06:18.285341+0000","last_undegraded":"2023-09-29T23:06:18.285341+0000","last_fullsized":"2023-09-29T23:06:18.285341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:20.907352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285659+0000","last_change":"2023-09-29T23:06:17.891510+0000","last_active":"2023-09-29T23:06:18.285659+0000","last_peered":"2023-09-29T23:06:18.285659+0000","last_clean":"2023-09-29T23:06:18.285659+0000","last_became_active":"2023-09-29T23:06:17.891217+0000","last_became_peered":"2023-09-29T23:06:17.891217+0000","last_unstale":"2023-09-29T23:06:18.285659+0000","last_undegraded":"2023-09-29T23:06:18.285659+0000","last_fullsized":"2023-09-29T23:06:18.285659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285386+0000","last_change":"2023-09-29T23:06:18.283220+0000","last_active":"2023-09-29T23:06:18.285386+0000","last_peered":"2023-09-29T23:06:18.285386+0000","last_clean":"2023-09-29T23:06:18.285386+0000","last_became_active":"2023-09-29T23:06:18.281619+0000","last_became_peered":"2023-09-29T23:06:18.281619+0000","last_unstale":"2023-09-29T23:06:18.285386+0000","last_undegraded":"2023-09-29T23:06:18.285386+0000","last_fullsized":"2023-09-29T23:06:18.285386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.039742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.272636+0000","last_change":"2023-09-29T23:06:17.887773+0000","last_active":"2023-09-29T23:06:18.272636+0000","last_peered":"2023-09-29T23:06:18.272636+0000","last_clean":"2023-09-29T23:06:18.272636+0000","last_became_active":"2023-09-29T23:06:17.887578+0000","last_became_peered":"2023-09-29T23:06:17.887578+0000","last_unstale":"2023-09-29T23:06:18.272636+0000","last_undegraded":"2023-09-29T23:06:18.272636+0000","last_fullsized":"2023-09-29T23:06:18.272636+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:04:22.787417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288175+0000","last_change":"2023-09-29T23:06:18.287213+0000","last_active":"2023-09-29T23:06:18.288175+0000","last_peered":"2023-09-29T23:06:18.288175+0000","last_clean":"2023-09-29T23:06:18.288175+0000","last_became_active":"2023-09-29T23:06:18.286861+0000","last_became_peered":"2023-09-29T23:06:18.286861+0000","last_unstale":"2023-09-29T23:06:18.288175+0000","last_undegraded":"2023-09-29T23:06:18.288175+0000","last_fullsized":"2023-09-29T23:06:18.288175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:44:31.777462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286577+0000","last_change":"2023-09-29T23:06:17.893529+0000","last_active":"2023-09-29T23:06:18.286577+0000","last_peered":"2023-09-29T23:06:18.286577+0000","last_clean":"2023-09-29T23:06:18.286577+0000","last_became_active":"2023-09-29T23:06:17.893269+0000","last_became_peered":"2023-09-29T23:06:17.893269+0000","last_unstale":"2023-09-29T23:06:18.286577+0000","last_undegraded":"2023-09-29T23:06:18.286577+0000","last_fullsized":"2023-09-29T23:06:18.286577+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288800+0000","last_change":"2023-09-29T23:06:18.286214+0000","last_active":"2023-09-29T23:06:18.288800+0000","last_peered":"2023-09-29T23:06:18.288800+0000","last_clean":"2023-09-29T23:06:18.288800+0000","last_became_active":"2023-09-29T23:06:18.285773+0000","last_became_peered":"2023-09-29T23:06:18.285773+0000","last_unstale":"2023-09-29T23:06:18.288800+0000","last_undegraded":"2023-09-29T23:06:18.288800+0000","last_fullsized":"2023-09-29T23:06:18.288800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278612+0000","last_change":"2023-09-29T23:06:17.892342+0000","last_active":"2023-09-29T23:06:18.278612+0000","last_peered":"2023-09-29T23:06:18.278612+0000","last_clean":"2023-09-29T23:06:18.278612+0000","last_became_active":"2023-09-29T23:06:17.891454+0000","last_became_peered":"2023-09-29T23:06:17.891454+0000","last_unstale":"2023-09-29T23:06:18.278612+0000","last_undegraded":"2023-09-29T23:06:18.278612+0000","last_fullsized":"2023-09-29T23:06:18.278612+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283064+0000","last_change":"2023-09-29T23:06:18.280840+0000","last_active":"2023-09-29T23:06:18.283064+0000","last_peered":"2023-09-29T23:06:18.283064+0000","last_clean":"2023-09-29T23:06:18.283064+0000","last_became_active":"2023-09-29T23:06:18.280656+0000","last_became_peered":"2023-09-29T23:06:18.280656+0000","last_unstale":"2023-09-29T23:06:18.283064+0000","last_undegraded":"2023-09-29T23:06:18.283064+0000","last_fullsized":"2023-09-29T23:06:18.283064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:13.688293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273870+0000","last_change":"2023-09-29T23:06:18.272838+0000","last_active":"2023-09-29T23:06:18.273870+0000","last_peered":"2023-09-29T23:06:18.273870+0000","last_clean":"2023-09-29T23:06:18.273870+0000","last_became_active":"2023-09-29T23:06:18.272444+0000","last_became_peered":"2023-09-29T23:06:18.272444+0000","last_unstale":"2023-09-29T23:06:18.273870+0000","last_undegraded":"2023-09-29T23:06:18.273870+0000","last_fullsized":"2023-09-29T23:06:18.273870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288812+0000","last_change":"2023-09-29T23:06:18.286345+0000","last_active":"2023-09-29T23:06:18.288812+0000","last_peered":"2023-09-29T23:06:18.288812+0000","last_clean":"2023-09-29T23:06:18.288812+0000","last_became_active":"2023-09-29T23:06:18.285985+0000","last_became_peered":"2023-09-29T23:06:18.285985+0000","last_unstale":"2023-09-29T23:06:18.288812+0000","last_undegraded":"2023-09-29T23:06:18.288812+0000","last_fullsized":"2023-09-29T23:06:18.288812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280311+0000","last_change":"2023-09-29T23:06:18.271411+0000","last_active":"2023-09-29T23:06:18.280311+0000","last_peered":"2023-09-29T23:06:18.280311+0000","last_clean":"2023-09-29T23:06:18.280311+0000","last_became_active":"2023-09-29T23:06:18.270318+0000","last_became_peered":"2023-09-29T23:06:18.270318+0000","last_unstale":"2023-09-29T23:06:18.280311+0000","last_undegraded":"2023-09-29T23:06:18.280311+0000","last_fullsized":"2023-09-29T23:06:18.280311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292425+0000","last_change":"2023-09-29T23:06:18.289646+0000","last_active":"2023-09-29T23:06:18.292425+0000","last_peered":"2023-09-29T23:06:18.292425+0000","last_clean":"2023-09-29T23:06:18.292425+0000","last_became_active":"2023-09-29T23:06:18.289304+0000","last_became_peered":"2023-09-29T23:06:18.289304+0000","last_unstale":"2023-09-29T23:06:18.292425+0000","last_undegraded":"2023-09-29T23:06:18.292425+0000","last_fullsized":"2023-09-29T23:06:18.292425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:18:01.223023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283682+0000","last_change":"2023-09-29T23:06:18.277502+0000","last_active":"2023-09-29T23:06:18.283682+0000","last_peered":"2023-09-29T23:06:18.283682+0000","last_clean":"2023-09-29T23:06:18.283682+0000","last_became_active":"2023-09-29T23:06:18.277224+0000","last_became_peered":"2023-09-29T23:06:18.277224+0000","last_unstale":"2023-09-29T23:06:18.283682+0000","last_undegraded":"2023-09-29T23:06:18.283682+0000","last_fullsized":"2023-09-29T23:06:18.283682+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":188416,"data_stored":60567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8960,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362880,"statfs":{"total":96636764160,"available":96627589120,"internally_reserved":0,"allocated":253952,"data_stored":652921,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8588,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363252,"statfs":{"total":96636764160,"available":96627970048,"internally_reserved":0,"allocated":200704,"data_stored":62035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9076,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362764,"statfs":{"total":96636764160,"available":96627470336,"internally_reserved":0,"allocated":241664,"data_stored":651317,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:06:26.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:06:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.947+0000 7fe61d6cd700 1 Processor -- start 2023-09-29T23:06:26.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.947+0000 7fe61d6cd700 1 -- start start 2023-09-29T23:06:26.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe61d6cd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6180a3cd0 0x7fe6180a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:26.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe61d6cd700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618063700 con 0x7fe61809f820 2023-09-29T23:06:26.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe61d6cd700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180a0880 con 0x7fe618063330 2023-09-29T23:06:26.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe61d6cd700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180a57e0 con 0x7fe6180a3cd0 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe6177fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6180a3cd0 0x7fe6180a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe6167fc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe61809f820 0x7fe6180a2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44428/0 (socket says 172.21.15.121:44428) 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe6167fc700 1 -- 172.21.15.121:0/2667175934 learned_addr learned my addr 172.21.15.121:0/2667175934 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe615ffb700 1 -- 172.21.15.121:0/2667175934 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772306111 0 0) 0x7fe618063700 con 0x7fe61809f820 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe6177fe700 1 -- 172.21.15.121:0/2667175934 >> v1:172.21.15.123:6789/0 conn(0x7fe618063330 legacy=0x7fe6180a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe6177fe700 1 -- 172.21.15.121:0/2667175934 >> v1:172.21.15.121:6789/0 conn(0x7fe61809f820 legacy=0x7fe6180a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe6177fe700 1 -- 172.21.15.121:0/2667175934 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6180a4250 con 0x7fe6180a3cd0 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.948+0000 7fe6177fe700 1 --2- 172.21.15.121:0/2667175934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6180a3cd0 0x7fe6180a2a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe610009e90 tx=0x7fe610009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b5fbddfe7d6e07b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe615ffb700 1 -- 172.21.15.121:0/2667175934 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000e070 con 0x7fe6180a3cd0 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe615ffb700 1 -- 172.21.15.121:0/2667175934 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61000bc30 con 0x7fe6180a3cd0 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe615ffb700 1 -- 172.21.15.121:0/2667175934 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610012430 con 0x7fe6180a3cd0 2023-09-29T23:06:26.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2667175934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6180a3cd0 msgr2=0x7fe6180a2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:26.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2667175934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6180a3cd0 0x7fe6180a2a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe610009e90 tx=0x7fe610009780 comp rx=0 tx=0).stop 2023-09-29T23:06:26.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2667175934 shutdown_connections 2023-09-29T23:06:26.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2667175934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6180a3cd0 0x7fe6180a2a70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:26.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2667175934 >> 172.21.15.121:0/2667175934 conn(0x7fe6180a5bc0 msgr2=0x7fe6180b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:26.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2667175934 shutdown_connections 2023-09-29T23:06:26.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.949+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2667175934 wait complete. 2023-09-29T23:06:26.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.950+0000 7fe61d6cd700 1 Processor -- start 2023-09-29T23:06:26.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.950+0000 7fe61d6cd700 1 -- start start 2023-09-29T23:06:26.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe61d6cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe618063330 0x7fe6181518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:26.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe61d6cd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe61809f820 0x7fe618095a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:26.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe61d6cd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6180a3cd0 0x7fe618095f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:26.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe61d6cd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180992f0 con 0x7fe618063330 2023-09-29T23:06:26.954 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe61d6cd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181522b0 con 0x7fe6180a3cd0 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe61d6cd700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe618152570 con 0x7fe61809f820 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe61809f820 0x7fe618095a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe616ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe618063330 0x7fe6181518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe61809f820 0x7fe618095a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36560/0 (socket says 172.21.15.121:36560) 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 -- 172.21.15.121:0/2834746118 learned_addr learned my addr 172.21.15.121:0/2834746118 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6177fe700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6180a3cd0 0x7fe618095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 -- 172.21.15.121:0/2834746118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6180a3cd0 msgr2=0x7fe618095f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6180a3cd0 0x7fe618095f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 -- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe618063330 msgr2=0x7fe6181518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe618063330 0x7fe6181518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 -- 172.21.15.121:0/2834746118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe610009a20 con 0x7fe61809f820 2023-09-29T23:06:26.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6177fe700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6180a3cd0 0x7fe618095f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe616ffd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe618063330 0x7fe6181518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.951+0000 7fe6167fc700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe61809f820 0x7fe618095a10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c810 tx=0x7fe608007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.952+0000 7fe5feffd700 1 -- 172.21.15.121:0/2834746118 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608005c00 con 0x7fe61809f820 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.952+0000 7fe5feffd700 1 -- 172.21.15.121:0/2834746118 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe608005d60 con 0x7fe61809f820 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.952+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61809a900 con 0x7fe61809f820 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.952+0000 7fe5feffd700 1 -- 172.21.15.121:0/2834746118 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608007320 con 0x7fe61809f820 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.952+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe618099f50 con 0x7fe61809f820 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.953+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61809a410 con 0x7fe61809f820 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.953+0000 7fe5feffd700 1 -- 172.21.15.121:0/2834746118 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe60801b3b0 con 0x7fe61809f820 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.953+0000 7fe5feffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:26.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.954+0000 7fe5feffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:26.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.954+0000 7fe5feffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:26.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.954+0000 7fe5feffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:26.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.954+0000 7fe5feffd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6040987b0 0x7fe60409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:26.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.954+0000 7fe616ffd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6040987b0 0x7fe60409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:26.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.954+0000 7fe5feffd700 1 -- 172.21.15.121:0/2834746118 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fe6080bf970 con 0x7fe61809f820 2023-09-29T23:06:26.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.955+0000 7fe616ffd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6040987b0 0x7fe60409ac60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe60c007fd0 tx=0x7fe60c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:26.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:26.958+0000 7fe5feffd700 1 -- 172.21.15.121:0/2834746118 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe61809a410 con 0x7fe61809f820 2023-09-29T23:06:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.137+0000 7fe61d6cd700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-29T23:06:27.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.137+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 --> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe61809a410 con 0x7fe6040987b0 2023-09-29T23:06:27.143 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.144+0000 7fe5feffd700 1 -- 172.21.15.121:0/2834746118 <== mgr.4107 v2:172.21.15.121:6808/113227 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+269438 (secure 0 0 0) 0x7fe61809a410 con 0x7fe6040987b0 2023-09-29T23:06:27.144 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.144+0000 7fe5feffd700 20 mgrc handle_command_reply tid 0 r 0 2023-09-29T23:06:27.147 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 10 mgrc shutdown 2023-09-29T23:06:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6040987b0 msgr2=0x7fe60409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6040987b0 0x7fe60409ac60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe60c007fd0 tx=0x7fe60c009040 comp rx=0 tx=0).stop 2023-09-29T23:06:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe61809f820 msgr2=0x7fe618095a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe61809f820 0x7fe618095a10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c810 tx=0x7fe608007d90 comp rx=0 tx=0).stop 2023-09-29T23:06:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 shutdown_connections 2023-09-29T23:06:27.149 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6040987b0 0x7fe60409ac60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6180a3cd0 0x7fe618095f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe618063330 0x7fe6181518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 --2- 172.21.15.121:0/2834746118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe61809f820 0x7fe618095a10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 >> 172.21.15.121:0/2834746118 conn(0x7fe6180a5bc0 msgr2=0x7fe6180b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 shutdown_connections 2023-09-29T23:06:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.147+0000 7fe61d6cd700 1 -- 172.21.15.121:0/2834746118 wait complete. 2023-09-29T23:06:27.150 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2023-09-29T23:06:27.159 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-09-29T23:06:25.487397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":35136,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":377452224,"statfs":{"total":386547056640,"available":386511077376,"internally_reserved":0,"allocated":884736,"data_stored":1426840,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.617246"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278366+0000","last_change":"2023-09-29T23:06:18.276673+0000","last_active":"2023-09-29T23:06:18.278366+0000","last_peered":"2023-09-29T23:06:18.278366+0000","last_clean":"2023-09-29T23:06:18.278366+0000","last_became_active":"2023-09-29T23:06:18.276312+0000","last_became_peered":"2023-09-29T23:06:18.276312+0000","last_unstale":"2023-09-29T23:06:18.278366+0000","last_undegraded":"2023-09-29T23:06:18.278366+0000","last_fullsized":"2023-09-29T23:06:18.278366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:40.183958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283354+0000","last_change":"2023-09-29T23:06:18.282307+0000","last_active":"2023-09-29T23:06:18.283354+0000","last_peered":"2023-09-29T23:06:18.283354+0000","last_clean":"2023-09-29T23:06:18.283354+0000","last_became_active":"2023-09-29T23:06:18.282064+0000","last_became_peered":"2023-09-29T23:06:18.282064+0000","last_unstale":"2023-09-29T23:06:18.283354+0000","last_undegraded":"2023-09-29T23:06:18.283354+0000","last_fullsized":"2023-09-29T23:06:18.283354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:10:08.336407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283157+0000","last_change":"2023-09-29T23:06:18.281514+0000","last_active":"2023-09-29T23:06:18.283157+0000","last_peered":"2023-09-29T23:06:18.283157+0000","last_clean":"2023-09-29T23:06:18.283157+0000","last_became_active":"2023-09-29T23:06:18.281144+0000","last_became_peered":"2023-09-29T23:06:18.281144+0000","last_unstale":"2023-09-29T23:06:18.283157+0000","last_undegraded":"2023-09-29T23:06:18.283157+0000","last_fullsized":"2023-09-29T23:06:18.283157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290407+0000","last_change":"2023-09-29T23:06:18.287192+0000","last_active":"2023-09-29T23:06:18.290407+0000","last_peered":"2023-09-29T23:06:18.290407+0000","last_clean":"2023-09-29T23:06:18.290407+0000","last_became_active":"2023-09-29T23:06:18.286917+0000","last_became_peered":"2023-09-29T23:06:18.286917+0000","last_unstale":"2023-09-29T23:06:18.290407+0000","last_undegraded":"2023-09-29T23:06:18.290407+0000","last_fullsized":"2023-09-29T23:06:18.290407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292437+0000","last_change":"2023-09-29T23:06:18.289803+0000","last_active":"2023-09-29T23:06:18.292437+0000","last_peered":"2023-09-29T23:06:18.292437+0000","last_clean":"2023-09-29T23:06:18.292437+0000","last_became_active":"2023-09-29T23:06:18.289379+0000","last_became_peered":"2023-09-29T23:06:18.289379+0000","last_unstale":"2023-09-29T23:06:18.292437+0000","last_undegraded":"2023-09-29T23:06:18.292437+0000","last_fullsized":"2023-09-29T23:06:18.292437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287884+0000","last_change":"2023-09-29T23:06:18.285440+0000","last_active":"2023-09-29T23:06:18.287884+0000","last_peered":"2023-09-29T23:06:18.287884+0000","last_clean":"2023-09-29T23:06:18.287884+0000","last_became_active":"2023-09-29T23:06:18.285182+0000","last_became_peered":"2023-09-29T23:06:18.285182+0000","last_unstale":"2023-09-29T23:06:18.287884+0000","last_undegraded":"2023-09-29T23:06:18.287884+0000","last_fullsized":"2023-09-29T23:06:18.287884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:04:37.136451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288766+0000","last_change":"2023-09-29T23:06:18.286007+0000","last_active":"2023-09-29T23:06:18.288766+0000","last_peered":"2023-09-29T23:06:18.288766+0000","last_clean":"2023-09-29T23:06:18.288766+0000","last_became_active":"2023-09-29T23:06:18.284765+0000","last_became_peered":"2023-09-29T23:06:18.284765+0000","last_unstale":"2023-09-29T23:06:18.288766+0000","last_undegraded":"2023-09-29T23:06:18.288766+0000","last_fullsized":"2023-09-29T23:06:18.288766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273887+0000","last_change":"2023-09-29T23:06:18.272963+0000","last_active":"2023-09-29T23:06:18.273887+0000","last_peered":"2023-09-29T23:06:18.273887+0000","last_clean":"2023-09-29T23:06:18.273887+0000","last_became_active":"2023-09-29T23:06:18.272736+0000","last_became_peered":"2023-09-29T23:06:18.272736+0000","last_unstale":"2023-09-29T23:06:18.273887+0000","last_undegraded":"2023-09-29T23:06:18.273887+0000","last_fullsized":"2023-09-29T23:06:18.273887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278347+0000","last_change":"2023-09-29T23:06:18.276308+0000","last_active":"2023-09-29T23:06:18.278347+0000","last_peered":"2023-09-29T23:06:18.278347+0000","last_clean":"2023-09-29T23:06:18.278347+0000","last_became_active":"2023-09-29T23:06:18.276056+0000","last_became_peered":"2023-09-29T23:06:18.276056+0000","last_unstale":"2023-09-29T23:06:18.278347+0000","last_undegraded":"2023-09-29T23:06:18.278347+0000","last_fullsized":"2023-09-29T23:06:18.278347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:36.204055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287088+0000","last_change":"2023-09-29T23:06:18.280125+0000","last_active":"2023-09-29T23:06:18.287088+0000","last_peered":"2023-09-29T23:06:18.287088+0000","last_clean":"2023-09-29T23:06:18.287088+0000","last_became_active":"2023-09-29T23:06:18.279830+0000","last_became_peered":"2023-09-29T23:06:18.279830+0000","last_unstale":"2023-09-29T23:06:18.287088+0000","last_undegraded":"2023-09-29T23:06:18.287088+0000","last_fullsized":"2023-09-29T23:06:18.287088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280206+0000","last_change":"2023-09-29T23:06:18.270148+0000","last_active":"2023-09-29T23:06:18.280206+0000","last_peered":"2023-09-29T23:06:18.280206+0000","last_clean":"2023-09-29T23:06:18.280206+0000","last_became_active":"2023-09-29T23:06:18.269859+0000","last_became_peered":"2023-09-29T23:06:18.269859+0000","last_unstale":"2023-09-29T23:06:18.280206+0000","last_undegraded":"2023-09-29T23:06:18.280206+0000","last_fullsized":"2023-09-29T23:06:18.280206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277323+0000","last_change":"2023-09-29T23:06:18.272855+0000","last_active":"2023-09-29T23:06:18.277323+0000","last_peered":"2023-09-29T23:06:18.277323+0000","last_clean":"2023-09-29T23:06:18.277323+0000","last_became_active":"2023-09-29T23:06:18.272591+0000","last_became_peered":"2023-09-29T23:06:18.272591+0000","last_unstale":"2023-09-29T23:06:18.277323+0000","last_undegraded":"2023-09-29T23:06:18.277323+0000","last_fullsized":"2023-09-29T23:06:18.277323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:35.444407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285404+0000","last_change":"2023-09-29T23:06:18.283392+0000","last_active":"2023-09-29T23:06:18.285404+0000","last_peered":"2023-09-29T23:06:18.285404+0000","last_clean":"2023-09-29T23:06:18.285404+0000","last_became_active":"2023-09-29T23:06:18.282197+0000","last_became_peered":"2023-09-29T23:06:18.282197+0000","last_unstale":"2023-09-29T23:06:18.285404+0000","last_undegraded":"2023-09-29T23:06:18.285404+0000","last_fullsized":"2023-09-29T23:06:18.285404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:33.387327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269370+0000","last_change":"2023-09-29T23:06:18.268361+0000","last_active":"2023-09-29T23:06:18.269370+0000","last_peered":"2023-09-29T23:06:18.269370+0000","last_clean":"2023-09-29T23:06:18.269370+0000","last_became_active":"2023-09-29T23:06:18.268085+0000","last_became_peered":"2023-09-29T23:06:18.268085+0000","last_unstale":"2023-09-29T23:06:18.269370+0000","last_undegraded":"2023-09-29T23:06:18.269370+0000","last_fullsized":"2023-09-29T23:06:18.269370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283344+0000","last_change":"2023-09-29T23:06:18.282177+0000","last_active":"2023-09-29T23:06:18.283344+0000","last_peered":"2023-09-29T23:06:18.283344+0000","last_clean":"2023-09-29T23:06:18.283344+0000","last_became_active":"2023-09-29T23:06:18.281841+0000","last_became_peered":"2023-09-29T23:06:18.281841+0000","last_unstale":"2023-09-29T23:06:18.283344+0000","last_undegraded":"2023-09-29T23:06:18.283344+0000","last_fullsized":"2023-09-29T23:06:18.283344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273852+0000","last_change":"2023-09-29T23:06:18.272693+0000","last_active":"2023-09-29T23:06:18.273852+0000","last_peered":"2023-09-29T23:06:18.273852+0000","last_clean":"2023-09-29T23:06:18.273852+0000","last_became_active":"2023-09-29T23:06:18.272433+0000","last_became_peered":"2023-09-29T23:06:18.272433+0000","last_unstale":"2023-09-29T23:06:18.273852+0000","last_undegraded":"2023-09-29T23:06:18.273852+0000","last_fullsized":"2023-09-29T23:06:18.273852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280760+0000","last_change":"2023-09-29T23:06:18.275372+0000","last_active":"2023-09-29T23:06:18.280760+0000","last_peered":"2023-09-29T23:06:18.280760+0000","last_clean":"2023-09-29T23:06:18.280760+0000","last_became_active":"2023-09-29T23:06:18.274861+0000","last_became_peered":"2023-09-29T23:06:18.274861+0000","last_unstale":"2023-09-29T23:06:18.280760+0000","last_undegraded":"2023-09-29T23:06:18.280760+0000","last_fullsized":"2023-09-29T23:06:18.280760+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290310+0000","last_change":"2023-09-29T23:06:18.287098+0000","last_active":"2023-09-29T23:06:18.290310+0000","last_peered":"2023-09-29T23:06:18.290310+0000","last_clean":"2023-09-29T23:06:18.290310+0000","last_became_active":"2023-09-29T23:06:18.286847+0000","last_became_peered":"2023-09-29T23:06:18.286847+0000","last_unstale":"2023-09-29T23:06:18.290310+0000","last_undegraded":"2023-09-29T23:06:18.290310+0000","last_fullsized":"2023-09-29T23:06:18.290310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277206+0000","last_change":"2023-09-29T23:06:18.273575+0000","last_active":"2023-09-29T23:06:18.277206+0000","last_peered":"2023-09-29T23:06:18.277206+0000","last_clean":"2023-09-29T23:06:18.277206+0000","last_became_active":"2023-09-29T23:06:18.273327+0000","last_became_peered":"2023-09-29T23:06:18.273327+0000","last_unstale":"2023-09-29T23:06:18.277206+0000","last_undegraded":"2023-09-29T23:06:18.277206+0000","last_fullsized":"2023-09-29T23:06:18.277206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:29:05.706848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280438+0000","last_change":"2023-09-29T23:06:18.271980+0000","last_active":"2023-09-29T23:06:18.280438+0000","last_peered":"2023-09-29T23:06:18.280438+0000","last_clean":"2023-09-29T23:06:18.280438+0000","last_became_active":"2023-09-29T23:06:18.271725+0000","last_became_peered":"2023-09-29T23:06:18.271725+0000","last_unstale":"2023-09-29T23:06:18.280438+0000","last_undegraded":"2023-09-29T23:06:18.280438+0000","last_fullsized":"2023-09-29T23:06:18.280438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283120+0000","last_change":"2023-09-29T23:06:18.281173+0000","last_active":"2023-09-29T23:06:18.283120+0000","last_peered":"2023-09-29T23:06:18.283120+0000","last_clean":"2023-09-29T23:06:18.283120+0000","last_became_active":"2023-09-29T23:06:18.280834+0000","last_became_peered":"2023-09-29T23:06:18.280834+0000","last_unstale":"2023-09-29T23:06:18.283120+0000","last_undegraded":"2023-09-29T23:06:18.283120+0000","last_fullsized":"2023-09-29T23:06:18.283120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:02:54.551488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280253+0000","last_change":"2023-09-29T23:06:18.270925+0000","last_active":"2023-09-29T23:06:18.280253+0000","last_peered":"2023-09-29T23:06:18.280253+0000","last_clean":"2023-09-29T23:06:18.280253+0000","last_became_active":"2023-09-29T23:06:18.269605+0000","last_became_peered":"2023-09-29T23:06:18.269605+0000","last_unstale":"2023-09-29T23:06:18.280253+0000","last_undegraded":"2023-09-29T23:06:18.280253+0000","last_fullsized":"2023-09-29T23:06:18.280253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287559+0000","last_change":"2023-09-29T23:06:18.284188+0000","last_active":"2023-09-29T23:06:18.287559+0000","last_peered":"2023-09-29T23:06:18.287559+0000","last_clean":"2023-09-29T23:06:18.287559+0000","last_became_active":"2023-09-29T23:06:18.283875+0000","last_became_peered":"2023-09-29T23:06:18.283875+0000","last_unstale":"2023-09-29T23:06:18.287559+0000","last_undegraded":"2023-09-29T23:06:18.287559+0000","last_fullsized":"2023-09-29T23:06:18.287559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283044+0000","last_change":"2023-09-29T23:06:18.280077+0000","last_active":"2023-09-29T23:06:18.283044+0000","last_peered":"2023-09-29T23:06:18.283044+0000","last_clean":"2023-09-29T23:06:18.283044+0000","last_became_active":"2023-09-29T23:06:18.279865+0000","last_became_peered":"2023-09-29T23:06:18.279865+0000","last_unstale":"2023-09-29T23:06:18.283044+0000","last_undegraded":"2023-09-29T23:06:18.283044+0000","last_fullsized":"2023-09-29T23:06:18.283044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:48.644161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280240+0000","last_change":"2023-09-29T23:06:18.270238+0000","last_active":"2023-09-29T23:06:18.280240+0000","last_peered":"2023-09-29T23:06:18.280240+0000","last_clean":"2023-09-29T23:06:18.280240+0000","last_became_active":"2023-09-29T23:06:18.269996+0000","last_became_peered":"2023-09-29T23:06:18.269996+0000","last_unstale":"2023-09-29T23:06:18.280240+0000","last_undegraded":"2023-09-29T23:06:18.280240+0000","last_fullsized":"2023-09-29T23:06:18.280240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287050+0000","last_change":"2023-09-29T23:06:18.279595+0000","last_active":"2023-09-29T23:06:18.287050+0000","last_peered":"2023-09-29T23:06:18.287050+0000","last_clean":"2023-09-29T23:06:18.287050+0000","last_became_active":"2023-09-29T23:06:18.278827+0000","last_became_peered":"2023-09-29T23:06:18.278827+0000","last_unstale":"2023-09-29T23:06:18.287050+0000","last_undegraded":"2023-09-29T23:06:18.287050+0000","last_fullsized":"2023-09-29T23:06:18.287050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288146+0000","last_change":"2023-09-29T23:06:18.287081+0000","last_active":"2023-09-29T23:06:18.288146+0000","last_peered":"2023-09-29T23:06:18.288146+0000","last_clean":"2023-09-29T23:06:18.288146+0000","last_became_active":"2023-09-29T23:06:18.286853+0000","last_became_peered":"2023-09-29T23:06:18.286853+0000","last_unstale":"2023-09-29T23:06:18.288146+0000","last_undegraded":"2023-09-29T23:06:18.288146+0000","last_fullsized":"2023-09-29T23:06:18.288146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:00.791418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280662+0000","last_change":"2023-09-29T23:06:17.886342+0000","last_active":"2023-09-29T23:06:18.280662+0000","last_peered":"2023-09-29T23:06:18.280662+0000","last_clean":"2023-09-29T23:06:18.280662+0000","last_became_active":"2023-09-29T23:06:17.886089+0000","last_became_peered":"2023-09-29T23:06:17.886089+0000","last_unstale":"2023-09-29T23:06:18.280662+0000","last_undegraded":"2023-09-29T23:06:18.280662+0000","last_fullsized":"2023-09-29T23:06:18.280662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:04.152750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288614+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288301+0000","last_became_peered":"2023-09-29T23:06:18.288301+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261493+0000","last_change":"2023-09-29T23:06:17.891522+0000","last_active":"2023-09-29T23:06:18.261493+0000","last_peered":"2023-09-29T23:06:18.261493+0000","last_clean":"2023-09-29T23:06:18.261493+0000","last_became_active":"2023-09-29T23:06:17.891232+0000","last_became_peered":"2023-09-29T23:06:17.891232+0000","last_unstale":"2023-09-29T23:06:18.261493+0000","last_undegraded":"2023-09-29T23:06:18.261493+0000","last_fullsized":"2023-09-29T23:06:18.261493+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287070+0000","last_change":"2023-09-29T23:06:18.279774+0000","last_active":"2023-09-29T23:06:18.287070+0000","last_peered":"2023-09-29T23:06:18.287070+0000","last_clean":"2023-09-29T23:06:18.287070+0000","last_became_active":"2023-09-29T23:06:18.279014+0000","last_became_peered":"2023-09-29T23:06:18.279014+0000","last_unstale":"2023-09-29T23:06:18.287070+0000","last_undegraded":"2023-09-29T23:06:18.287070+0000","last_fullsized":"2023-09-29T23:06:18.287070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273886+0000","last_change":"2023-09-29T23:06:17.894216+0000","last_active":"2023-09-29T23:06:18.273886+0000","last_peered":"2023-09-29T23:06:18.273886+0000","last_clean":"2023-09-29T23:06:18.273886+0000","last_became_active":"2023-09-29T23:06:17.893923+0000","last_became_peered":"2023-09-29T23:06:17.893923+0000","last_unstale":"2023-09-29T23:06:18.273886+0000","last_undegraded":"2023-09-29T23:06:18.273886+0000","last_fullsized":"2023-09-29T23:06:18.273886+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282454+0000","last_change":"2023-09-29T23:06:18.276320+0000","last_active":"2023-09-29T23:06:18.282454+0000","last_peered":"2023-09-29T23:06:18.282454+0000","last_clean":"2023-09-29T23:06:18.282454+0000","last_became_active":"2023-09-29T23:06:18.276049+0000","last_became_peered":"2023-09-29T23:06:18.276049+0000","last_unstale":"2023-09-29T23:06:18.282454+0000","last_undegraded":"2023-09-29T23:06:18.282454+0000","last_fullsized":"2023-09-29T23:06:18.282454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:32.688280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261797+0000","last_change":"2023-09-29T23:06:17.883845+0000","last_active":"2023-09-29T23:06:18.261797+0000","last_peered":"2023-09-29T23:06:18.261797+0000","last_clean":"2023-09-29T23:06:18.261797+0000","last_became_active":"2023-09-29T23:06:17.883559+0000","last_became_peered":"2023-09-29T23:06:17.883559+0000","last_unstale":"2023-09-29T23:06:18.261797+0000","last_undegraded":"2023-09-29T23:06:18.261797+0000","last_fullsized":"2023-09-29T23:06:18.261797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286481+0000","last_change":"2023-09-29T23:06:18.284038+0000","last_active":"2023-09-29T23:06:18.286481+0000","last_peered":"2023-09-29T23:06:18.286481+0000","last_clean":"2023-09-29T23:06:18.286481+0000","last_became_active":"2023-09-29T23:06:18.283764+0000","last_became_peered":"2023-09-29T23:06:18.283764+0000","last_unstale":"2023-09-29T23:06:18.286481+0000","last_undegraded":"2023-09-29T23:06:18.286481+0000","last_fullsized":"2023-09-29T23:06:18.286481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:05:42.345774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261094+0000","last_change":"2023-09-29T23:06:17.886414+0000","last_active":"2023-09-29T23:06:18.261094+0000","last_peered":"2023-09-29T23:06:18.261094+0000","last_clean":"2023-09-29T23:06:18.261094+0000","last_became_active":"2023-09-29T23:06:17.886124+0000","last_became_peered":"2023-09-29T23:06:17.886124+0000","last_unstale":"2023-09-29T23:06:18.261094+0000","last_undegraded":"2023-09-29T23:06:18.261094+0000","last_fullsized":"2023-09-29T23:06:18.261094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:32.385528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288784+0000","last_change":"2023-09-29T23:06:18.286164+0000","last_active":"2023-09-29T23:06:18.288784+0000","last_peered":"2023-09-29T23:06:18.288784+0000","last_clean":"2023-09-29T23:06:18.288784+0000","last_became_active":"2023-09-29T23:06:18.285518+0000","last_became_peered":"2023-09-29T23:06:18.285518+0000","last_unstale":"2023-09-29T23:06:18.288784+0000","last_undegraded":"2023-09-29T23:06:18.288784+0000","last_fullsized":"2023-09-29T23:06:18.288784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:54:18.859428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286862+0000","last_change":"2023-09-29T23:06:17.889196+0000","last_active":"2023-09-29T23:06:18.286862+0000","last_peered":"2023-09-29T23:06:18.286862+0000","last_clean":"2023-09-29T23:06:18.286862+0000","last_became_active":"2023-09-29T23:06:17.888932+0000","last_became_peered":"2023-09-29T23:06:17.888932+0000","last_unstale":"2023-09-29T23:06:18.286862+0000","last_undegraded":"2023-09-29T23:06:18.286862+0000","last_fullsized":"2023-09-29T23:06:18.286862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280276+0000","last_change":"2023-09-29T23:06:18.271028+0000","last_active":"2023-09-29T23:06:18.280276+0000","last_peered":"2023-09-29T23:06:18.280276+0000","last_clean":"2023-09-29T23:06:18.280276+0000","last_became_active":"2023-09-29T23:06:18.270012+0000","last_became_peered":"2023-09-29T23:06:18.270012+0000","last_unstale":"2023-09-29T23:06:18.280276+0000","last_undegraded":"2023-09-29T23:06:18.280276+0000","last_fullsized":"2023-09-29T23:06:18.280276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.293088+0000","last_change":"2023-09-29T23:06:17.889224+0000","last_active":"2023-09-29T23:06:18.293088+0000","last_peered":"2023-09-29T23:06:18.293088+0000","last_clean":"2023-09-29T23:06:18.293088+0000","last_became_active":"2023-09-29T23:06:17.888941+0000","last_became_peered":"2023-09-29T23:06:17.888941+0000","last_unstale":"2023-09-29T23:06:18.293088+0000","last_undegraded":"2023-09-29T23:06:18.293088+0000","last_fullsized":"2023-09-29T23:06:18.293088+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:06:09.687876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274555+0000","last_change":"2023-09-29T23:06:18.270435+0000","last_active":"2023-09-29T23:06:18.274555+0000","last_peered":"2023-09-29T23:06:18.274555+0000","last_clean":"2023-09-29T23:06:18.274555+0000","last_became_active":"2023-09-29T23:06:18.268172+0000","last_became_peered":"2023-09-29T23:06:18.268172+0000","last_unstale":"2023-09-29T23:06:18.274555+0000","last_undegraded":"2023-09-29T23:06:18.274555+0000","last_fullsized":"2023-09-29T23:06:18.274555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:45.768259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289061+0000","last_change":"2023-09-29T23:06:17.893695+0000","last_active":"2023-09-29T23:06:18.289061+0000","last_peered":"2023-09-29T23:06:18.289061+0000","last_clean":"2023-09-29T23:06:18.289061+0000","last_became_active":"2023-09-29T23:06:17.893381+0000","last_became_peered":"2023-09-29T23:06:17.893381+0000","last_unstale":"2023-09-29T23:06:18.289061+0000","last_undegraded":"2023-09-29T23:06:18.289061+0000","last_fullsized":"2023-09-29T23:06:18.289061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292208+0000","last_change":"2023-09-29T23:06:18.287485+0000","last_active":"2023-09-29T23:06:18.292208+0000","last_peered":"2023-09-29T23:06:18.292208+0000","last_clean":"2023-09-29T23:06:18.292208+0000","last_became_active":"2023-09-29T23:06:18.285220+0000","last_became_peered":"2023-09-29T23:06:18.285220+0000","last_unstale":"2023-09-29T23:06:18.292208+0000","last_undegraded":"2023-09-29T23:06:18.292208+0000","last_fullsized":"2023-09-29T23:06:18.292208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260745+0000","last_change":"2023-09-29T23:06:17.888841+0000","last_active":"2023-09-29T23:06:18.260745+0000","last_peered":"2023-09-29T23:06:18.260745+0000","last_clean":"2023-09-29T23:06:18.260745+0000","last_became_active":"2023-09-29T23:06:17.888515+0000","last_became_peered":"2023-09-29T23:06:17.888515+0000","last_unstale":"2023-09-29T23:06:18.260745+0000","last_undegraded":"2023-09-29T23:06:18.260745+0000","last_fullsized":"2023-09-29T23:06:18.260745+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287868+0000","last_change":"2023-09-29T23:06:18.285208+0000","last_active":"2023-09-29T23:06:18.287868+0000","last_peered":"2023-09-29T23:06:18.287868+0000","last_clean":"2023-09-29T23:06:18.287868+0000","last_became_active":"2023-09-29T23:06:18.284879+0000","last_became_peered":"2023-09-29T23:06:18.284879+0000","last_unstale":"2023-09-29T23:06:18.287868+0000","last_undegraded":"2023-09-29T23:06:18.287868+0000","last_fullsized":"2023-09-29T23:06:18.287868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260580+0000","last_change":"2023-09-29T23:06:17.890315+0000","last_active":"2023-09-29T23:06:18.260580+0000","last_peered":"2023-09-29T23:06:18.260580+0000","last_clean":"2023-09-29T23:06:18.260580+0000","last_became_active":"2023-09-29T23:06:17.890017+0000","last_became_peered":"2023-09-29T23:06:17.890017+0000","last_unstale":"2023-09-29T23:06:18.260580+0000","last_undegraded":"2023-09-29T23:06:18.260580+0000","last_fullsized":"2023-09-29T23:06:18.260580+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:55:35.860921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288529+0000","last_change":"2023-09-29T23:06:18.285771+0000","last_active":"2023-09-29T23:06:18.288529+0000","last_peered":"2023-09-29T23:06:18.288529+0000","last_clean":"2023-09-29T23:06:18.288529+0000","last_became_active":"2023-09-29T23:06:18.285493+0000","last_became_peered":"2023-09-29T23:06:18.285493+0000","last_unstale":"2023-09-29T23:06:18.288529+0000","last_undegraded":"2023-09-29T23:06:18.288529+0000","last_fullsized":"2023-09-29T23:06:18.288529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282000+0000","last_change":"2023-09-29T23:06:17.891835+0000","last_active":"2023-09-29T23:06:18.282000+0000","last_peered":"2023-09-29T23:06:18.282000+0000","last_clean":"2023-09-29T23:06:18.282000+0000","last_became_active":"2023-09-29T23:06:17.891563+0000","last_became_peered":"2023-09-29T23:06:17.891563+0000","last_unstale":"2023-09-29T23:06:18.282000+0000","last_undegraded":"2023-09-29T23:06:18.282000+0000","last_fullsized":"2023-09-29T23:06:18.282000+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:22.597416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292423+0000","last_change":"2023-09-29T23:06:18.289343+0000","last_active":"2023-09-29T23:06:18.292423+0000","last_peered":"2023-09-29T23:06:18.292423+0000","last_clean":"2023-09-29T23:06:18.292423+0000","last_became_active":"2023-09-29T23:06:18.289072+0000","last_became_peered":"2023-09-29T23:06:18.289072+0000","last_unstale":"2023-09-29T23:06:18.292423+0000","last_undegraded":"2023-09-29T23:06:18.292423+0000","last_fullsized":"2023-09-29T23:06:18.292423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:27:12.781947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282222+0000","last_change":"2023-09-29T23:06:17.891844+0000","last_active":"2023-09-29T23:06:18.282222+0000","last_peered":"2023-09-29T23:06:18.282222+0000","last_clean":"2023-09-29T23:06:18.282222+0000","last_became_active":"2023-09-29T23:06:17.891582+0000","last_became_peered":"2023-09-29T23:06:17.891582+0000","last_unstale":"2023-09-29T23:06:18.282222+0000","last_undegraded":"2023-09-29T23:06:18.282222+0000","last_fullsized":"2023-09-29T23:06:18.282222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:59.037943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287898+0000","last_change":"2023-09-29T23:06:18.285944+0000","last_active":"2023-09-29T23:06:18.287898+0000","last_peered":"2023-09-29T23:06:18.287898+0000","last_clean":"2023-09-29T23:06:18.287898+0000","last_became_active":"2023-09-29T23:06:18.285661+0000","last_became_peered":"2023-09-29T23:06:18.285661+0000","last_unstale":"2023-09-29T23:06:18.287898+0000","last_undegraded":"2023-09-29T23:06:18.287898+0000","last_fullsized":"2023-09-29T23:06:18.287898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:20:25.540032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.275819+0000","last_change":"2023-09-29T23:06:17.889165+0000","last_active":"2023-09-29T23:06:18.275819+0000","last_peered":"2023-09-29T23:06:18.275819+0000","last_clean":"2023-09-29T23:06:18.275819+0000","last_became_active":"2023-09-29T23:06:17.888842+0000","last_became_peered":"2023-09-29T23:06:17.888842+0000","last_unstale":"2023-09-29T23:06:18.275819+0000","last_undegraded":"2023-09-29T23:06:18.275819+0000","last_fullsized":"2023-09-29T23:06:18.275819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:35.202255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287772+0000","last_change":"2023-09-29T23:06:18.277402+0000","last_active":"2023-09-29T23:06:18.287772+0000","last_peered":"2023-09-29T23:06:18.287772+0000","last_clean":"2023-09-29T23:06:18.287772+0000","last_became_active":"2023-09-29T23:06:18.277159+0000","last_became_peered":"2023-09-29T23:06:18.277159+0000","last_unstale":"2023-09-29T23:06:18.287772+0000","last_undegraded":"2023-09-29T23:06:18.287772+0000","last_fullsized":"2023-09-29T23:06:18.287772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273008+0000","last_change":"2023-09-29T23:06:17.890237+0000","last_active":"2023-09-29T23:06:18.273008+0000","last_peered":"2023-09-29T23:06:18.273008+0000","last_clean":"2023-09-29T23:06:18.273008+0000","last_became_active":"2023-09-29T23:06:17.889971+0000","last_became_peered":"2023-09-29T23:06:17.889971+0000","last_unstale":"2023-09-29T23:06:18.273008+0000","last_undegraded":"2023-09-29T23:06:18.273008+0000","last_fullsized":"2023-09-29T23:06:18.273008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:57.168709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282466+0000","last_change":"2023-09-29T23:06:18.276262+0000","last_active":"2023-09-29T23:06:18.282466+0000","last_peered":"2023-09-29T23:06:18.282466+0000","last_clean":"2023-09-29T23:06:18.282466+0000","last_became_active":"2023-09-29T23:06:18.276040+0000","last_became_peered":"2023-09-29T23:06:18.276040+0000","last_unstale":"2023-09-29T23:06:18.282466+0000","last_undegraded":"2023-09-29T23:06:18.282466+0000","last_fullsized":"2023-09-29T23:06:18.282466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260885+0000","last_change":"2023-09-29T23:06:17.888828+0000","last_active":"2023-09-29T23:06:18.260885+0000","last_peered":"2023-09-29T23:06:18.260885+0000","last_clean":"2023-09-29T23:06:18.260885+0000","last_became_active":"2023-09-29T23:06:17.888525+0000","last_became_peered":"2023-09-29T23:06:17.888525+0000","last_unstale":"2023-09-29T23:06:18.260885+0000","last_undegraded":"2023-09-29T23:06:18.260885+0000","last_fullsized":"2023-09-29T23:06:18.260885+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:18:04.884961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285419+0000","last_change":"2023-09-29T23:06:18.283530+0000","last_active":"2023-09-29T23:06:18.285419+0000","last_peered":"2023-09-29T23:06:18.285419+0000","last_clean":"2023-09-29T23:06:18.285419+0000","last_became_active":"2023-09-29T23:06:18.281248+0000","last_became_peered":"2023-09-29T23:06:18.281248+0000","last_unstale":"2023-09-29T23:06:18.285419+0000","last_undegraded":"2023-09-29T23:06:18.285419+0000","last_fullsized":"2023-09-29T23:06:18.285419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:23.599215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289625+0000","last_change":"2023-09-29T23:06:17.894026+0000","last_active":"2023-09-29T23:06:18.289625+0000","last_peered":"2023-09-29T23:06:18.289625+0000","last_clean":"2023-09-29T23:06:18.289625+0000","last_became_active":"2023-09-29T23:06:17.893757+0000","last_became_peered":"2023-09-29T23:06:17.893757+0000","last_unstale":"2023-09-29T23:06:18.289625+0000","last_undegraded":"2023-09-29T23:06:18.289625+0000","last_fullsized":"2023-09-29T23:06:18.289625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286633+0000","last_change":"2023-09-29T23:06:18.278135+0000","last_active":"2023-09-29T23:06:18.286633+0000","last_peered":"2023-09-29T23:06:18.286633+0000","last_clean":"2023-09-29T23:06:18.286633+0000","last_became_active":"2023-09-29T23:06:18.277865+0000","last_became_peered":"2023-09-29T23:06:18.277865+0000","last_unstale":"2023-09-29T23:06:18.286633+0000","last_undegraded":"2023-09-29T23:06:18.286633+0000","last_fullsized":"2023-09-29T23:06:18.286633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260719+0000","last_change":"2023-09-29T23:06:17.891680+0000","last_active":"2023-09-29T23:06:18.260719+0000","last_peered":"2023-09-29T23:06:18.260719+0000","last_clean":"2023-09-29T23:06:18.260719+0000","last_became_active":"2023-09-29T23:06:17.891413+0000","last_became_peered":"2023-09-29T23:06:17.891413+0000","last_unstale":"2023-09-29T23:06:18.260719+0000","last_undegraded":"2023-09-29T23:06:18.260719+0000","last_fullsized":"2023-09-29T23:06:18.260719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:19.905553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269364+0000","last_change":"2023-09-29T23:06:18.268288+0000","last_active":"2023-09-29T23:06:18.269364+0000","last_peered":"2023-09-29T23:06:18.269364+0000","last_clean":"2023-09-29T23:06:18.269364+0000","last_became_active":"2023-09-29T23:06:18.268079+0000","last_became_peered":"2023-09-29T23:06:18.268079+0000","last_unstale":"2023-09-29T23:06:18.269364+0000","last_undegraded":"2023-09-29T23:06:18.269364+0000","last_fullsized":"2023-09-29T23:06:18.269364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274260+0000","last_change":"2023-09-29T23:06:17.892193+0000","last_active":"2023-09-29T23:06:18.274260+0000","last_peered":"2023-09-29T23:06:18.274260+0000","last_clean":"2023-09-29T23:06:18.274260+0000","last_became_active":"2023-09-29T23:06:17.891898+0000","last_became_peered":"2023-09-29T23:06:17.891898+0000","last_unstale":"2023-09-29T23:06:18.274260+0000","last_undegraded":"2023-09-29T23:06:18.274260+0000","last_fullsized":"2023-09-29T23:06:18.274260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.276612+0000","last_change":"2023-09-29T23:06:18.271513+0000","last_active":"2023-09-29T23:06:18.276612+0000","last_peered":"2023-09-29T23:06:18.276612+0000","last_clean":"2023-09-29T23:06:18.276612+0000","last_became_active":"2023-09-29T23:06:18.271257+0000","last_became_peered":"2023-09-29T23:06:18.271257+0000","last_unstale":"2023-09-29T23:06:18.276612+0000","last_undegraded":"2023-09-29T23:06:18.276612+0000","last_fullsized":"2023-09-29T23:06:18.276612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:23.301482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273504+0000","last_change":"2023-09-29T23:06:17.893616+0000","last_active":"2023-09-29T23:06:18.273504+0000","last_peered":"2023-09-29T23:06:18.273504+0000","last_clean":"2023-09-29T23:06:18.273504+0000","last_became_active":"2023-09-29T23:06:17.893285+0000","last_became_peered":"2023-09-29T23:06:17.893285+0000","last_unstale":"2023-09-29T23:06:18.273504+0000","last_undegraded":"2023-09-29T23:06:18.273504+0000","last_fullsized":"2023-09-29T23:06:18.273504+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283690+0000","last_change":"2023-09-29T23:06:18.277473+0000","last_active":"2023-09-29T23:06:18.283690+0000","last_peered":"2023-09-29T23:06:18.283690+0000","last_clean":"2023-09-29T23:06:18.283690+0000","last_became_active":"2023-09-29T23:06:18.276953+0000","last_became_peered":"2023-09-29T23:06:18.276953+0000","last_unstale":"2023-09-29T23:06:18.283690+0000","last_undegraded":"2023-09-29T23:06:18.283690+0000","last_fullsized":"2023-09-29T23:06:18.283690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264133+0000","last_change":"2023-09-29T23:06:13.639723+0000","last_active":"2023-09-29T23:06:18.264133+0000","last_peered":"2023-09-29T23:06:18.264133+0000","last_clean":"2023-09-29T23:06:18.264133+0000","last_became_active":"2023-09-29T23:06:13.639406+0000","last_became_peered":"2023-09-29T23:06:13.639406+0000","last_unstale":"2023-09-29T23:06:18.264133+0000","last_undegraded":"2023-09-29T23:06:18.264133+0000","last_fullsized":"2023-09-29T23:06:18.264133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:07:23.007465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261412+0000","last_change":"2023-09-29T23:06:17.894258+0000","last_active":"2023-09-29T23:06:18.261412+0000","last_peered":"2023-09-29T23:06:18.261412+0000","last_clean":"2023-09-29T23:06:18.261412+0000","last_became_active":"2023-09-29T23:06:17.893935+0000","last_became_peered":"2023-09-29T23:06:17.893935+0000","last_unstale":"2023-09-29T23:06:18.261412+0000","last_undegraded":"2023-09-29T23:06:18.261412+0000","last_fullsized":"2023-09-29T23:06:18.261412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:30:41.674282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280296+0000","last_change":"2023-09-29T23:06:18.271186+0000","last_active":"2023-09-29T23:06:18.280296+0000","last_peered":"2023-09-29T23:06:18.280296+0000","last_clean":"2023-09-29T23:06:18.280296+0000","last_became_active":"2023-09-29T23:06:18.270718+0000","last_became_peered":"2023-09-29T23:06:18.270718+0000","last_unstale":"2023-09-29T23:06:18.280296+0000","last_undegraded":"2023-09-29T23:06:18.280296+0000","last_fullsized":"2023-09-29T23:06:18.280296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:58.170509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260686+0000","last_change":"2023-09-29T23:06:13.380282+0000","last_active":"2023-09-29T23:06:18.260686+0000","last_peered":"2023-09-29T23:06:18.260686+0000","last_clean":"2023-09-29T23:06:18.260686+0000","last_became_active":"2023-09-29T23:06:13.379948+0000","last_became_peered":"2023-09-29T23:06:13.379948+0000","last_unstale":"2023-09-29T23:06:18.260686+0000","last_undegraded":"2023-09-29T23:06:18.260686+0000","last_fullsized":"2023-09-29T23:06:18.260686+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:02.759420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261279+0000","last_change":"2023-09-29T23:06:17.884199+0000","last_active":"2023-09-29T23:06:18.261279+0000","last_peered":"2023-09-29T23:06:18.261279+0000","last_clean":"2023-09-29T23:06:18.261279+0000","last_became_active":"2023-09-29T23:06:17.883932+0000","last_became_peered":"2023-09-29T23:06:17.883932+0000","last_unstale":"2023-09-29T23:06:18.261279+0000","last_undegraded":"2023-09-29T23:06:18.261279+0000","last_fullsized":"2023-09-29T23:06:18.261279+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:39.182159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283703+0000","last_change":"2023-09-29T23:06:18.277498+0000","last_active":"2023-09-29T23:06:18.283703+0000","last_peered":"2023-09-29T23:06:18.283703+0000","last_clean":"2023-09-29T23:06:18.283703+0000","last_became_active":"2023-09-29T23:06:18.276885+0000","last_became_peered":"2023-09-29T23:06:18.276885+0000","last_unstale":"2023-09-29T23:06:18.283703+0000","last_undegraded":"2023-09-29T23:06:18.283703+0000","last_fullsized":"2023-09-29T23:06:18.283703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.259675+0000","last_change":"2023-09-29T23:06:13.639753+0000","last_active":"2023-09-29T23:06:18.259675+0000","last_peered":"2023-09-29T23:06:18.259675+0000","last_clean":"2023-09-29T23:06:18.259675+0000","last_became_active":"2023-09-29T23:06:13.639436+0000","last_became_peered":"2023-09-29T23:06:13.639436+0000","last_unstale":"2023-09-29T23:06:18.259675+0000","last_undegraded":"2023-09-29T23:06:18.259675+0000","last_fullsized":"2023-09-29T23:06:18.259675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:09:57.060849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261100+0000","last_change":"2023-09-29T23:06:17.885616+0000","last_active":"2023-09-29T23:06:18.261100+0000","last_peered":"2023-09-29T23:06:18.261100+0000","last_clean":"2023-09-29T23:06:18.261100+0000","last_became_active":"2023-09-29T23:06:17.885342+0000","last_became_peered":"2023-09-29T23:06:17.885342+0000","last_unstale":"2023-09-29T23:06:18.261100+0000","last_undegraded":"2023-09-29T23:06:18.261100+0000","last_fullsized":"2023-09-29T23:06:18.261100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280779+0000","last_change":"2023-09-29T23:06:18.277223+0000","last_active":"2023-09-29T23:06:18.280779+0000","last_peered":"2023-09-29T23:06:18.280779+0000","last_clean":"2023-09-29T23:06:18.280779+0000","last_became_active":"2023-09-29T23:06:18.276548+0000","last_became_peered":"2023-09-29T23:06:18.276548+0000","last_unstale":"2023-09-29T23:06:18.280779+0000","last_undegraded":"2023-09-29T23:06:18.280779+0000","last_fullsized":"2023-09-29T23:06:18.280779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261249+0000","last_change":"2023-09-29T23:06:13.380967+0000","last_active":"2023-09-29T23:06:18.261249+0000","last_peered":"2023-09-29T23:06:18.261249+0000","last_clean":"2023-09-29T23:06:18.261249+0000","last_became_active":"2023-09-29T23:06:13.379890+0000","last_became_peered":"2023-09-29T23:06:13.379890+0000","last_unstale":"2023-09-29T23:06:18.261249+0000","last_undegraded":"2023-09-29T23:06:18.261249+0000","last_fullsized":"2023-09-29T23:06:18.261249+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:46.738824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260986+0000","last_change":"2023-09-29T23:06:09.717533+0000","last_active":"2023-09-29T23:06:18.260986+0000","last_peered":"2023-09-29T23:06:18.260986+0000","last_clean":"2023-09-29T23:06:18.260986+0000","last_became_active":"2023-09-29T23:06:09.716536+0000","last_became_peered":"2023-09-29T23:06:09.716536+0000","last_unstale":"2023-09-29T23:06:18.260986+0000","last_undegraded":"2023-09-29T23:06:18.260986+0000","last_fullsized":"2023-09-29T23:06:18.260986+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_clean_scrub_stamp":"2023-09-29T23:06:08.290533+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:44:31.128783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274816+0000","last_change":"2023-09-29T23:06:17.892420+0000","last_active":"2023-09-29T23:06:18.274816+0000","last_peered":"2023-09-29T23:06:18.274816+0000","last_clean":"2023-09-29T23:06:18.274816+0000","last_became_active":"2023-09-29T23:06:17.892121+0000","last_became_peered":"2023-09-29T23:06:17.892121+0000","last_unstale":"2023-09-29T23:06:18.274816+0000","last_undegraded":"2023-09-29T23:06:18.274816+0000","last_fullsized":"2023-09-29T23:06:18.274816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283140+0000","last_change":"2023-09-29T23:06:18.281364+0000","last_active":"2023-09-29T23:06:18.283140+0000","last_peered":"2023-09-29T23:06:18.283140+0000","last_clean":"2023-09-29T23:06:18.283140+0000","last_became_active":"2023-09-29T23:06:18.280821+0000","last_became_peered":"2023-09-29T23:06:18.280821+0000","last_unstale":"2023-09-29T23:06:18.283140+0000","last_undegraded":"2023-09-29T23:06:18.283140+0000","last_fullsized":"2023-09-29T23:06:18.283140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260488+0000","last_change":"2023-09-29T23:06:13.380209+0000","last_active":"2023-09-29T23:06:18.260488+0000","last_peered":"2023-09-29T23:06:18.260488+0000","last_clean":"2023-09-29T23:06:18.260488+0000","last_became_active":"2023-09-29T23:06:13.379845+0000","last_became_peered":"2023-09-29T23:06:13.379845+0000","last_unstale":"2023-09-29T23:06:18.260488+0000","last_undegraded":"2023-09-29T23:06:18.260488+0000","last_fullsized":"2023-09-29T23:06:18.260488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261391+0000","last_change":"2023-09-29T23:06:17.891971+0000","last_active":"2023-09-29T23:06:18.261391+0000","last_peered":"2023-09-29T23:06:18.261391+0000","last_clean":"2023-09-29T23:06:18.261391+0000","last_became_active":"2023-09-29T23:06:17.891695+0000","last_became_peered":"2023-09-29T23:06:17.891695+0000","last_unstale":"2023-09-29T23:06:18.261391+0000","last_undegraded":"2023-09-29T23:06:18.261391+0000","last_fullsized":"2023-09-29T23:06:18.261391+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288649+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288351+0000","last_became_peered":"2023-09-29T23:06:18.288351+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261483+0000","last_change":"2023-09-29T23:06:13.380613+0000","last_active":"2023-09-29T23:06:18.261483+0000","last_peered":"2023-09-29T23:06:18.261483+0000","last_clean":"2023-09-29T23:06:18.261483+0000","last_became_active":"2023-09-29T23:06:13.380347+0000","last_became_peered":"2023-09-29T23:06:13.380347+0000","last_unstale":"2023-09-29T23:06:18.261483+0000","last_undegraded":"2023-09-29T23:06:18.261483+0000","last_fullsized":"2023-09-29T23:06:18.261483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:35.568211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264019+0000","last_change":"2023-09-29T23:06:17.885489+0000","last_active":"2023-09-29T23:06:18.264019+0000","last_peered":"2023-09-29T23:06:18.264019+0000","last_clean":"2023-09-29T23:06:18.264019+0000","last_became_active":"2023-09-29T23:06:17.884966+0000","last_became_peered":"2023-09-29T23:06:17.884966+0000","last_unstale":"2023-09-29T23:06:18.264019+0000","last_undegraded":"2023-09-29T23:06:18.264019+0000","last_fullsized":"2023-09-29T23:06:18.264019+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286897+0000","last_change":"2023-09-29T23:06:18.282836+0000","last_active":"2023-09-29T23:06:18.286897+0000","last_peered":"2023-09-29T23:06:18.286897+0000","last_clean":"2023-09-29T23:06:18.286897+0000","last_became_active":"2023-09-29T23:06:18.282215+0000","last_became_peered":"2023-09-29T23:06:18.282215+0000","last_unstale":"2023-09-29T23:06:18.286897+0000","last_undegraded":"2023-09-29T23:06:18.286897+0000","last_fullsized":"2023-09-29T23:06:18.286897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288992+0000","last_change":"2023-09-29T23:06:17.894025+0000","last_active":"2023-09-29T23:06:18.288992+0000","last_peered":"2023-09-29T23:06:18.288992+0000","last_clean":"2023-09-29T23:06:18.288992+0000","last_became_active":"2023-09-29T23:06:17.893701+0000","last_became_peered":"2023-09-29T23:06:17.893701+0000","last_unstale":"2023-09-29T23:06:18.288992+0000","last_undegraded":"2023-09-29T23:06:18.288992+0000","last_fullsized":"2023-09-29T23:06:18.288992+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.270574+0000","last_change":"2023-09-29T23:06:13.379718+0000","last_active":"2023-09-29T23:06:18.270574+0000","last_peered":"2023-09-29T23:06:18.270574+0000","last_clean":"2023-09-29T23:06:18.270574+0000","last_became_active":"2023-09-29T23:06:13.379450+0000","last_became_peered":"2023-09-29T23:06:13.379450+0000","last_unstale":"2023-09-29T23:06:18.270574+0000","last_undegraded":"2023-09-29T23:06:18.270574+0000","last_fullsized":"2023-09-29T23:06:18.270574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286470+0000","last_change":"2023-09-29T23:06:18.283988+0000","last_active":"2023-09-29T23:06:18.286470+0000","last_peered":"2023-09-29T23:06:18.286470+0000","last_clean":"2023-09-29T23:06:18.286470+0000","last_became_active":"2023-09-29T23:06:18.283448+0000","last_became_peered":"2023-09-29T23:06:18.283448+0000","last_unstale":"2023-09-29T23:06:18.286470+0000","last_undegraded":"2023-09-29T23:06:18.286470+0000","last_fullsized":"2023-09-29T23:06:18.286470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260973+0000","last_change":"2023-09-29T23:06:13.382829+0000","last_active":"2023-09-29T23:06:18.260973+0000","last_peered":"2023-09-29T23:06:18.260973+0000","last_clean":"2023-09-29T23:06:18.260973+0000","last_became_active":"2023-09-29T23:06:13.382590+0000","last_became_peered":"2023-09-29T23:06:13.382590+0000","last_unstale":"2023-09-29T23:06:18.260973+0000","last_undegraded":"2023-09-29T23:06:18.260973+0000","last_fullsized":"2023-09-29T23:06:18.260973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:06.822425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.271596+0000","last_change":"2023-09-29T23:06:17.881636+0000","last_active":"2023-09-29T23:06:18.271596+0000","last_peered":"2023-09-29T23:06:18.271596+0000","last_clean":"2023-09-29T23:06:18.271596+0000","last_became_active":"2023-09-29T23:06:17.881343+0000","last_became_peered":"2023-09-29T23:06:17.881343+0000","last_unstale":"2023-09-29T23:06:18.271596+0000","last_undegraded":"2023-09-29T23:06:18.271596+0000","last_fullsized":"2023-09-29T23:06:18.271596+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274553+0000","last_change":"2023-09-29T23:06:18.268514+0000","last_active":"2023-09-29T23:06:18.274553+0000","last_peered":"2023-09-29T23:06:18.274553+0000","last_clean":"2023-09-29T23:06:18.274553+0000","last_became_active":"2023-09-29T23:06:18.268089+0000","last_became_peered":"2023-09-29T23:06:18.268089+0000","last_unstale":"2023-09-29T23:06:18.274553+0000","last_undegraded":"2023-09-29T23:06:18.274553+0000","last_fullsized":"2023-09-29T23:06:18.274553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:22.930009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288913+0000","last_change":"2023-09-29T23:06:17.892888+0000","last_active":"2023-09-29T23:06:18.288913+0000","last_peered":"2023-09-29T23:06:18.288913+0000","last_clean":"2023-09-29T23:06:18.288913+0000","last_became_active":"2023-09-29T23:06:17.892666+0000","last_became_peered":"2023-09-29T23:06:17.892666+0000","last_unstale":"2023-09-29T23:06:18.288913+0000","last_undegraded":"2023-09-29T23:06:18.288913+0000","last_fullsized":"2023-09-29T23:06:18.288913+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285341+0000","last_change":"2023-09-29T23:06:18.282028+0000","last_active":"2023-09-29T23:06:18.285341+0000","last_peered":"2023-09-29T23:06:18.285341+0000","last_clean":"2023-09-29T23:06:18.285341+0000","last_became_active":"2023-09-29T23:06:18.281546+0000","last_became_peered":"2023-09-29T23:06:18.281546+0000","last_unstale":"2023-09-29T23:06:18.285341+0000","last_undegraded":"2023-09-29T23:06:18.285341+0000","last_fullsized":"2023-09-29T23:06:18.285341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:20.907352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285659+0000","last_change":"2023-09-29T23:06:17.891510+0000","last_active":"2023-09-29T23:06:18.285659+0000","last_peered":"2023-09-29T23:06:18.285659+0000","last_clean":"2023-09-29T23:06:18.285659+0000","last_became_active":"2023-09-29T23:06:17.891217+0000","last_became_peered":"2023-09-29T23:06:17.891217+0000","last_unstale":"2023-09-29T23:06:18.285659+0000","last_undegraded":"2023-09-29T23:06:18.285659+0000","last_fullsized":"2023-09-29T23:06:18.285659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285386+0000","last_change":"2023-09-29T23:06:18.283220+0000","last_active":"2023-09-29T23:06:18.285386+0000","last_peered":"2023-09-29T23:06:18.285386+0000","last_clean":"2023-09-29T23:06:18.285386+0000","last_became_active":"2023-09-29T23:06:18.281619+0000","last_became_peered":"2023-09-29T23:06:18.281619+0000","last_unstale":"2023-09-29T23:06:18.285386+0000","last_undegraded":"2023-09-29T23:06:18.285386+0000","last_fullsized":"2023-09-29T23:06:18.285386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.039742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.272636+0000","last_change":"2023-09-29T23:06:17.887773+0000","last_active":"2023-09-29T23:06:18.272636+0000","last_peered":"2023-09-29T23:06:18.272636+0000","last_clean":"2023-09-29T23:06:18.272636+0000","last_became_active":"2023-09-29T23:06:17.887578+0000","last_became_peered":"2023-09-29T23:06:17.887578+0000","last_unstale":"2023-09-29T23:06:18.272636+0000","last_undegraded":"2023-09-29T23:06:18.272636+0000","last_fullsized":"2023-09-29T23:06:18.272636+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:04:22.787417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288175+0000","last_change":"2023-09-29T23:06:18.287213+0000","last_active":"2023-09-29T23:06:18.288175+0000","last_peered":"2023-09-29T23:06:18.288175+0000","last_clean":"2023-09-29T23:06:18.288175+0000","last_became_active":"2023-09-29T23:06:18.286861+0000","last_became_peered":"2023-09-29T23:06:18.286861+0000","last_unstale":"2023-09-29T23:06:18.288175+0000","last_undegraded":"2023-09-29T23:06:18.288175+0000","last_fullsized":"2023-09-29T23:06:18.288175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:44:31.777462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286577+0000","last_change":"2023-09-29T23:06:17.893529+0000","last_active":"2023-09-29T23:06:18.286577+0000","last_peered":"2023-09-29T23:06:18.286577+0000","last_clean":"2023-09-29T23:06:18.286577+0000","last_became_active":"2023-09-29T23:06:17.893269+0000","last_became_peered":"2023-09-29T23:06:17.893269+0000","last_unstale":"2023-09-29T23:06:18.286577+0000","last_undegraded":"2023-09-29T23:06:18.286577+0000","last_fullsized":"2023-09-29T23:06:18.286577+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288800+0000","last_change":"2023-09-29T23:06:18.286214+0000","last_active":"2023-09-29T23:06:18.288800+0000","last_peered":"2023-09-29T23:06:18.288800+0000","last_clean":"2023-09-29T23:06:18.288800+0000","last_became_active":"2023-09-29T23:06:18.285773+0000","last_became_peered":"2023-09-29T23:06:18.285773+0000","last_unstale":"2023-09-29T23:06:18.288800+0000","last_undegraded":"2023-09-29T23:06:18.288800+0000","last_fullsized":"2023-09-29T23:06:18.288800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278612+0000","last_change":"2023-09-29T23:06:17.892342+0000","last_active":"2023-09-29T23:06:18.278612+0000","last_peered":"2023-09-29T23:06:18.278612+0000","last_clean":"2023-09-29T23:06:18.278612+0000","last_became_active":"2023-09-29T23:06:17.891454+0000","last_became_peered":"2023-09-29T23:06:17.891454+0000","last_unstale":"2023-09-29T23:06:18.278612+0000","last_undegraded":"2023-09-29T23:06:18.278612+0000","last_fullsized":"2023-09-29T23:06:18.278612+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283064+0000","last_change":"2023-09-29T23:06:18.280840+0000","last_active":"2023-09-29T23:06:18.283064+0000","last_peered":"2023-09-29T23:06:18.283064+0000","last_clean":"2023-09-29T23:06:18.283064+0000","last_became_active":"2023-09-29T23:06:18.280656+0000","last_became_peered":"2023-09-29T23:06:18.280656+0000","last_unstale":"2023-09-29T23:06:18.283064+0000","last_undegraded":"2023-09-29T23:06:18.283064+0000","last_fullsized":"2023-09-29T23:06:18.283064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:13.688293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273870+0000","last_change":"2023-09-29T23:06:18.272838+0000","last_active":"2023-09-29T23:06:18.273870+0000","last_peered":"2023-09-29T23:06:18.273870+0000","last_clean":"2023-09-29T23:06:18.273870+0000","last_became_active":"2023-09-29T23:06:18.272444+0000","last_became_peered":"2023-09-29T23:06:18.272444+0000","last_unstale":"2023-09-29T23:06:18.273870+0000","last_undegraded":"2023-09-29T23:06:18.273870+0000","last_fullsized":"2023-09-29T23:06:18.273870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288812+0000","last_change":"2023-09-29T23:06:18.286345+0000","last_active":"2023-09-29T23:06:18.288812+0000","last_peered":"2023-09-29T23:06:18.288812+0000","last_clean":"2023-09-29T23:06:18.288812+0000","last_became_active":"2023-09-29T23:06:18.285985+0000","last_became_peered":"2023-09-29T23:06:18.285985+0000","last_unstale":"2023-09-29T23:06:18.288812+0000","last_undegraded":"2023-09-29T23:06:18.288812+0000","last_fullsized":"2023-09-29T23:06:18.288812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280311+0000","last_change":"2023-09-29T23:06:18.271411+0000","last_active":"2023-09-29T23:06:18.280311+0000","last_peered":"2023-09-29T23:06:18.280311+0000","last_clean":"2023-09-29T23:06:18.280311+0000","last_became_active":"2023-09-29T23:06:18.270318+0000","last_became_peered":"2023-09-29T23:06:18.270318+0000","last_unstale":"2023-09-29T23:06:18.280311+0000","last_undegraded":"2023-09-29T23:06:18.280311+0000","last_fullsized":"2023-09-29T23:06:18.280311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292425+0000","last_change":"2023-09-29T23:06:18.289646+0000","last_active":"2023-09-29T23:06:18.292425+0000","last_peered":"2023-09-29T23:06:18.292425+0000","last_clean":"2023-09-29T23:06:18.292425+0000","last_became_active":"2023-09-29T23:06:18.289304+0000","last_became_peered":"2023-09-29T23:06:18.289304+0000","last_unstale":"2023-09-29T23:06:18.292425+0000","last_undegraded":"2023-09-29T23:06:18.292425+0000","last_fullsized":"2023-09-29T23:06:18.292425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:18:01.223023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283682+0000","last_change":"2023-09-29T23:06:18.277502+0000","last_active":"2023-09-29T23:06:18.283682+0000","last_peered":"2023-09-29T23:06:18.283682+0000","last_clean":"2023-09-29T23:06:18.283682+0000","last_became_active":"2023-09-29T23:06:18.277224+0000","last_became_peered":"2023-09-29T23:06:18.277224+0000","last_unstale":"2023-09-29T23:06:18.283682+0000","last_undegraded":"2023-09-29T23:06:18.283682+0000","last_fullsized":"2023-09-29T23:06:18.283682+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":188416,"data_stored":60567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8960,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362880,"statfs":{"total":96636764160,"available":96627589120,"internally_reserved":0,"allocated":253952,"data_stored":652921,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8588,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363252,"statfs":{"total":96636764160,"available":96627970048,"internally_reserved":0,"allocated":200704,"data_stored":62035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9076,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362764,"statfs":{"total":96636764160,"available":96627470336,"internally_reserved":0,"allocated":241664,"data_stored":651317,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:06:27.165 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-09-29T23:06:27.165 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-29T23:06:27.165 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-29T23:06:27.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-29T23:06:27.166 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-29T23:06:27.317 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.315+0000 7f645e59c700 1 Processor -- start 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f645e59c700 1 -- start start 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f645e59c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6458063330 0x7f64580a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f645e59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f645809f820 0x7f64580a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f645e59c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6458063700 con 0x7f64580a2360 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f645e59c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64580a57e0 con 0x7f6458063330 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f645e59c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64580a4040 con 0x7f645809f820 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f64577fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f645809f820 0x7f64580a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f64577fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f645809f820 0x7f64580a3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36562/0 (socket says 172.21.15.121:36562) 2023-09-29T23:06:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f64577fe700 1 -- 172.21.15.121:0/1475499308 learned_addr learned my addr 172.21.15.121:0/1475499308 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f6457fff700 1 --2- 172.21.15.121:0/1475499308 >> v2:172.21.15.123:3300/0 conn(0x7f6458063330 0x7f64580a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.316+0000 7f64577fe700 1 -- 172.21.15.121:0/1475499308 >> v2:172.21.15.123:3300/0 conn(0x7f6458063330 msgr2=0x7f64580a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f64577fe700 1 --2- 172.21.15.121:0/1475499308 >> v2:172.21.15.123:3300/0 conn(0x7f6458063330 0x7f64580a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f64577fe700 1 -- 172.21.15.121:0/1475499308 >> v1:172.21.15.123:6789/0 conn(0x7f64580a2360 legacy=0x7f64580a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f64577fe700 1 -- 172.21.15.121:0/1475499308 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64580a0880 con 0x7f645809f820 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f6457fff700 1 --2- 172.21.15.121:0/1475499308 >> v2:172.21.15.123:3300/0 conn(0x7f6458063330 0x7f64580a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f64577fe700 1 --2- 172.21.15.121:0/1475499308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f645809f820 0x7f64580a3b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f644c0099b0 tx=0x7f644c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb1e0a6b5607c8e2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f6456ffd700 1 -- 172.21.15.121:0/1475499308 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c00b820 con 0x7f645809f820 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f6456ffd700 1 -- 172.21.15.121:0/1475499308 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c00b980 con 0x7f645809f820 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f6456ffd700 1 -- 172.21.15.121:0/1475499308 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c00bd20 con 0x7f645809f820 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 -- 172.21.15.121:0/1475499308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f645809f820 msgr2=0x7f64580a3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 --2- 172.21.15.121:0/1475499308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f645809f820 0x7f64580a3b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f644c0099b0 tx=0x7f644c009250 comp rx=0 tx=0).stop 2023-09-29T23:06:27.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 -- 172.21.15.121:0/1475499308 shutdown_connections 2023-09-29T23:06:27.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 --2- 172.21.15.121:0/1475499308 >> v2:172.21.15.123:3300/0 conn(0x7f6458063330 0x7f64580a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 --2- 172.21.15.121:0/1475499308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f645809f820 0x7f64580a3b00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 -- 172.21.15.121:0/1475499308 >> 172.21.15.121:0/1475499308 conn(0x7f64580a5bc0 msgr2=0x7f64580b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:27.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 -- 172.21.15.121:0/1475499308 shutdown_connections 2023-09-29T23:06:27.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.317+0000 7f645e59c700 1 -- 172.21.15.121:0/1475499308 wait complete. 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.318+0000 7f645e59c700 1 Processor -- start 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.318+0000 7f645e59c700 1 -- start start 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6458063330 0x7f6458099bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645809f820 0x7f645806df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645e59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64580a2360 0x7f645813eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645e59c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645813f410 con 0x7f645809f820 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645e59c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64580967a0 con 0x7f6458063330 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645e59c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645806e8c0 con 0x7f64580a2360 2023-09-29T23:06:27.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64580a2360 0x7f645813eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f64577fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645809f820 0x7f645806df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64580a2360 0x7f645813eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36576/0 (socket says 172.21.15.121:36576) 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f64577fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645809f820 0x7f645806df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58512/0 (socket says 172.21.15.121:58512) 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 -- 172.21.15.121:0/1715957602 learned_addr learned my addr 172.21.15.121:0/1715957602 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f6457fff700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6458063330 0x7f6458099bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 -- 172.21.15.121:0/1715957602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6458063330 msgr2=0x7f6458099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6458063330 0x7f6458099bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 -- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645809f820 msgr2=0x7f645806df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645809f820 0x7f645806df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.319+0000 7f645cb39700 1 -- 172.21.15.121:0/1715957602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0094f0 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.320+0000 7f645cb39700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64580a2360 0x7f645813eb20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f645000bc30 tx=0x7f6450009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.320+0000 7f643ffff700 1 -- 172.21.15.121:0/1715957602 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000c840 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.320+0000 7f643ffff700 1 -- 172.21.15.121:0/1715957602 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645000c9e0 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.320+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6458095b10 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.320+0000 7f643ffff700 1 -- 172.21.15.121:0/1715957602 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000cda0 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.320+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64580960a0 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.321+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645806abd0 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f643ffff700 1 -- 172.21.15.121:0/1715957602 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6450014620 con 0x7f64580a2360 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f643ffff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:27.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f643ffff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:27.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f643ffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:27.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f643ffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:27.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f643ffff700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f64440987b0 0x7f644409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f6457fff700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f64440987b0 0x7f644409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f643ffff700 1 -- 172.21.15.121:0/1715957602 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f64500c02f0 con 0x7f64580a2360 2023-09-29T23:06:27.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.322+0000 7f6457fff700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f64440987b0 0x7f644409ac60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f644800ad40 tx=0x7f6448008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:27.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.326+0000 7f643ffff700 1 -- 172.21.15.121:0/1715957602 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f645008da30 con 0x7f64580a2360 2023-09-29T23:06:27.556 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.556+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f6458071660 con 0x7f64580a2360 2023-09-29T23:06:27.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.560+0000 7f643ffff700 1 -- 172.21.15.121:0/1715957602 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+244971 (secure 0 0 0) 0x7f6450091050 con 0x7f64580a2360 2023-09-29T23:06:27.563 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 10 mgrc shutdown 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f64440987b0 msgr2=0x7f644409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f64440987b0 0x7f644409ac60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f644800ad40 tx=0x7f6448008040 comp rx=0 tx=0).stop 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64580a2360 msgr2=0x7f645813eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64580a2360 0x7f645813eb20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f645000bc30 tx=0x7f6450009820 comp rx=0 tx=0).stop 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 shutdown_connections 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f64440987b0 0x7f644409ac60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64580a2360 0x7f645813eb20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6458063330 0x7f6458099bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 --2- 172.21.15.121:0/1715957602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f645809f820 0x7f645806df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 >> 172.21.15.121:0/1715957602 conn(0x7f64580a5bc0 msgr2=0x7f64580a34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:27.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 shutdown_connections 2023-09-29T23:06:27.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.563+0000 7f645e59c700 1 -- 172.21.15.121:0/1715957602 wait complete. 2023-09-29T23:06:27.575 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":5,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":113227},{"type":"v1","addr":"172.21.15.121:6809","nonce":113227}]},"active_addr":"172.21.15.121:6809/113227","active_change":"2023-09-29T23:06:01.479590+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":4117,"name":"z","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":3908926366}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":3281027898}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":324249002}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":1245941524}]}]} 2023-09-29T23:06:27.581 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-29T23:06:27.581 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-29T23:06:27.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:27.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.732+0000 7f32e3eb2700 1 Processor -- start 2023-09-29T23:06:27.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.732+0000 7f32e3eb2700 1 -- start start 2023-09-29T23:06:27.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e3eb2700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f32dc069d00 0x7f32dc06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e3eb2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f32dc13da80 0x7f32dc06e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e3eb2700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32dc06a0d0 con 0x7f32dc13dec0 2023-09-29T23:06:27.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e3eb2700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32dc0a5bc0 con 0x7f32dc069d00 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e3eb2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32dc06daa0 con 0x7f32dc13da80 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e1c4e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f32dc069d00 0x7f32dc06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e1c4e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f32dc069d00 0x7f32dc06d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58528/0 (socket says 172.21.15.121:58528) 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e244f700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f32dc13dec0 0x7f32dc140740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44444/0 (socket says 172.21.15.121:44444) 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e1c4e700 1 -- 172.21.15.121:0/1797859627 learned_addr learned my addr 172.21.15.121:0/1797859627 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e144d700 1 --2- 172.21.15.121:0/1797859627 >> v2:172.21.15.123:3300/0 conn(0x7f32dc13da80 0x7f32dc06e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e1c4e700 1 -- 172.21.15.121:0/1797859627 >> v2:172.21.15.123:3300/0 conn(0x7f32dc13da80 msgr2=0x7f32dc06e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e1c4e700 1 --2- 172.21.15.121:0/1797859627 >> v2:172.21.15.123:3300/0 conn(0x7f32dc13da80 0x7f32dc06e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e1c4e700 1 -- 172.21.15.121:0/1797859627 >> v1:172.21.15.121:6789/0 conn(0x7f32dc13dec0 legacy=0x7f32dc140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.733+0000 7f32e1c4e700 1 -- 172.21.15.121:0/1797859627 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc06f1f0 con 0x7f32dc069d00 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e1c4e700 1 --2- 172.21.15.121:0/1797859627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc069d00 0x7f32dc06d560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f32dc06b970 tx=0x7f32cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e704b59a304e4d5 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e0c4c700 1 -- 172.21.15.121:0/1797859627 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc00b860 con 0x7f32dc069d00 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e0c4c700 1 -- 172.21.15.121:0/1797859627 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc00b9c0 con 0x7f32dc069d00 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e0c4c700 1 -- 172.21.15.121:0/1797859627 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc00bd60 con 0x7f32dc069d00 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 -- 172.21.15.121:0/1797859627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc069d00 msgr2=0x7f32dc06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/1797859627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc069d00 0x7f32dc06d560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f32dc06b970 tx=0x7f32cc009250 comp rx=0 tx=0).stop 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 -- 172.21.15.121:0/1797859627 shutdown_connections 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/1797859627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc069d00 0x7f32dc06d560 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/1797859627 >> v2:172.21.15.123:3300/0 conn(0x7f32dc13da80 0x7f32dc06e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 -- 172.21.15.121:0/1797859627 >> 172.21.15.121:0/1797859627 conn(0x7f32dc068550 msgr2=0x7f32dc068950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:27.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 -- 172.21.15.121:0/1797859627 shutdown_connections 2023-09-29T23:06:27.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.734+0000 7f32e3eb2700 1 -- 172.21.15.121:0/1797859627 wait complete. 2023-09-29T23:06:27.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.735+0000 7f32e3eb2700 1 Processor -- start 2023-09-29T23:06:27.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.735+0000 7f32e3eb2700 1 -- start start 2023-09-29T23:06:27.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e3eb2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32dc069d00 0x7f32dc096020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e3eb2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc13da80 0x7f32dc097160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e3eb2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32dc13dec0 0x7f32dc13b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e3eb2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc097b50 con 0x7f32dc13da80 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e3eb2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc096c40 con 0x7f32dc13dec0 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e3eb2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc0969d0 con 0x7f32dc069d00 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e1c4e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32dc069d00 0x7f32dc096020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e144d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc13da80 0x7f32dc097160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e1c4e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32dc069d00 0x7f32dc096020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36582/0 (socket says 172.21.15.121:36582) 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e1c4e700 1 -- 172.21.15.121:0/926725107 learned_addr learned my addr 172.21.15.121:0/926725107 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e244f700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32dc13dec0 0x7f32dc13b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e144d700 1 -- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32dc069d00 msgr2=0x7f32dc096020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e144d700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32dc069d00 0x7f32dc096020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e144d700 1 -- 172.21.15.121:0/926725107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32dc13dec0 msgr2=0x7f32dc13b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e144d700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32dc13dec0 0x7f32dc13b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e144d700 1 -- 172.21.15.121:0/926725107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc0094f0 con 0x7f32dc13da80 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e244f700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32dc13dec0 0x7f32dc13b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32e144d700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc13da80 0x7f32dc097160 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f32d800d8c0 tx=0x7f32d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32d1ffb700 1 -- 172.21.15.121:0/926725107 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d80125f0 con 0x7f32dc13da80 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.736+0000 7f32d1ffb700 1 -- 172.21.15.121:0/926725107 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8012750 con 0x7f32dc13da80 2023-09-29T23:06:27.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.737+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32dc0b19b0 con 0x7f32dc13da80 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.737+0000 7f32d1ffb700 1 -- 172.21.15.121:0/926725107 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8012af0 con 0x7f32dc13da80 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.737+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32dc145550 con 0x7f32dc13da80 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.738+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32dc071660 con 0x7f32dc13da80 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32d1ffb700 1 -- 172.21.15.121:0/926725107 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f32d801e460 con 0x7f32dc13da80 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32d1ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32d1ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32d1ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32d1ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32d1ffb700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f32c0098800 0x7f32c009acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32e1c4e700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f32c0098800 0x7f32c009acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.739+0000 7f32d1ffb700 1 -- 172.21.15.121:0/926725107 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f32d80c0a60 con 0x7f32dc13da80 2023-09-29T23:06:27.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.740+0000 7f32e1c4e700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f32c0098800 0x7f32c009acb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009290 tx=0x7f32cc017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:27.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.744+0000 7f32d1ffb700 1 -- 172.21.15.121:0/926725107 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32d808e100 con 0x7f32dc13da80 2023-09-29T23:06:27.926 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.927+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32dc140610 con 0x7f32dc13da80 2023-09-29T23:06:27.927 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.927+0000 7f32d1ffb700 1 -- 172.21.15.121:0/926725107 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12254 (secure 0 0 0) 0x7f32d8091720 con 0x7f32dc13da80 2023-09-29T23:06:27.927 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:27.927 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":19,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:19.255324+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-29T23:06:16.382138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-29T23:06:17.218163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.929+0000 7f32e3eb2700 10 mgrc shutdown 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.929+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f32c0098800 msgr2=0x7f32c009acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.929+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f32c0098800 0x7f32c009acb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009290 tx=0x7f32cc017000 comp rx=0 tx=0).stop 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.929+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc13da80 msgr2=0x7f32dc097160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.929+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc13da80 0x7f32dc097160 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f32d800d8c0 tx=0x7f32d8008040 comp rx=0 tx=0).stop 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 shutdown_connections 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f32c0098800 0x7f32c009acb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32dc13dec0 0x7f32dc13b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32dc069d00 0x7f32dc096020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 --2- 172.21.15.121:0/926725107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32dc13da80 0x7f32dc097160 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 >> 172.21.15.121:0/926725107 conn(0x7f32dc068550 msgr2=0x7f32dc13e540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 shutdown_connections 2023-09-29T23:06:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:27.930+0000 7f32e3eb2700 1 -- 172.21.15.121:0/926725107 wait complete. 2023-09-29T23:06:27.941 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-29T23:06:27.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:06:28.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.058+0000 7f2c45724700 1 Processor -- start 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.058+0000 7f2c45724700 1 -- start start 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c45724700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c400630e0 0x7f2c40142000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c45724700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c400634b0 0x7f2c4009b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c45724700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c40142a60 con 0x7f2c40142540 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c45724700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c40142c90 con 0x7f2c400630e0 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c45724700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c40143f60 con 0x7f2c400634b0 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3effd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c400630e0 0x7f2c40142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3e7fc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2c40142540 0x7f2c4009ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44452/0 (socket says 172.21.15.121:44452) 2023-09-29T23:06:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3effd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c400630e0 0x7f2c40142000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58544/0 (socket says 172.21.15.121:58544) 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3e7fc700 1 -- 172.21.15.121:0/2492745214 learned_addr learned my addr 172.21.15.121:0/2492745214 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3f7fe700 1 --2- 172.21.15.121:0/2492745214 >> v2:172.21.15.123:3300/0 conn(0x7f2c400634b0 0x7f2c4009b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3effd700 1 -- 172.21.15.121:0/2492745214 >> v2:172.21.15.123:3300/0 conn(0x7f2c400634b0 msgr2=0x7f2c4009b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3effd700 1 --2- 172.21.15.121:0/2492745214 >> v2:172.21.15.123:3300/0 conn(0x7f2c400634b0 0x7f2c4009b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3effd700 1 -- 172.21.15.121:0/2492745214 >> v1:172.21.15.121:6789/0 conn(0x7f2c40142540 legacy=0x7f2c4009ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3effd700 1 -- 172.21.15.121:0/2492745214 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c401440d0 con 0x7f2c400630e0 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.059+0000 7f2c3dffb700 1 -- 172.21.15.121:0/2492745214 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455237656 0 0) 0x7f2c40142a60 con 0x7f2c40142540 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c3effd700 1 --2- 172.21.15.121:0/2492745214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c40142000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c4009a350 tx=0x7f2c34009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0b07f12042f9db server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c3dffb700 1 -- 172.21.15.121:0/2492745214 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34005c10 con 0x7f2c400630e0 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c3dffb700 1 -- 172.21.15.121:0/2492745214 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c34005d70 con 0x7f2c400630e0 2023-09-29T23:06:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c3dffb700 1 -- 172.21.15.121:0/2492745214 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34006260 con 0x7f2c400630e0 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 -- 172.21.15.121:0/2492745214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 msgr2=0x7f2c40142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 --2- 172.21.15.121:0/2492745214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c40142000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c4009a350 tx=0x7f2c34009120 comp rx=0 tx=0).stop 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 -- 172.21.15.121:0/2492745214 shutdown_connections 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 --2- 172.21.15.121:0/2492745214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c40142000 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 --2- 172.21.15.121:0/2492745214 >> v2:172.21.15.123:3300/0 conn(0x7f2c400634b0 0x7f2c4009b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 -- 172.21.15.121:0/2492745214 >> 172.21.15.121:0/2492745214 conn(0x7f2c400a5bc0 msgr2=0x7f2c400b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 -- 172.21.15.121:0/2492745214 shutdown_connections 2023-09-29T23:06:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.060+0000 7f2c45724700 1 -- 172.21.15.121:0/2492745214 wait complete. 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.061+0000 7f2c45724700 1 Processor -- start 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.061+0000 7f2c45724700 1 -- start start 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c45724700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c4009cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c45724700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400634b0 0x7f2c401360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c45724700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c40142540 0x7f2c40133fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c45724700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4006dd30 con 0x7f2c400630e0 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c45724700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c40136630 con 0x7f2c400634b0 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c45724700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c40134e00 con 0x7f2c40142540 2023-09-29T23:06:28.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c4009cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c4009cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58546/0 (socket says 172.21.15.121:58546) 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 -- 172.21.15.121:0/2283604923 learned_addr learned my addr 172.21.15.121:0/2283604923 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3f7fe700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c40142540 0x7f2c40133fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3e7fc700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400634b0 0x7f2c401360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 -- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c40142540 msgr2=0x7f2c40133fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c40142540 0x7f2c40133fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 -- 172.21.15.121:0/2283604923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400634b0 msgr2=0x7f2c401360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400634b0 0x7f2c401360f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 -- 172.21.15.121:0/2283604923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340093c0 con 0x7f2c400630e0 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.062+0000 7f2c3effd700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c4009cc10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c40064110 tx=0x7f2c34016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.063+0000 7f2c26ffd700 1 -- 172.21.15.121:0/2283604923 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3401d800 con 0x7f2c400630e0 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.063+0000 7f2c26ffd700 1 -- 172.21.15.121:0/2283604923 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3401d960 con 0x7f2c400630e0 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.063+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4006f1c0 con 0x7f2c400630e0 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.063+0000 7f2c26ffd700 1 -- 172.21.15.121:0/2283604923 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3401dd40 con 0x7f2c400630e0 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.063+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4006d990 con 0x7f2c400630e0 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.064+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c40134fc0 con 0x7f2c400630e0 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.065+0000 7f2c26ffd700 1 -- 172.21.15.121:0/2283604923 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2c3401b690 con 0x7f2c400630e0 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.065+0000 7f2c26ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.066+0000 7f2c26ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.066+0000 7f2c26ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.066+0000 7f2c26ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.066+0000 7f2c26ffd700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c2c098780 0x7f2c2c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.066+0000 7f2c3e7fc700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c2c098780 0x7f2c2c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.066+0000 7f2c26ffd700 1 -- 172.21.15.121:0/2283604923 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f2c340c24a0 con 0x7f2c400630e0 2023-09-29T23:06:28.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.066+0000 7f2c3e7fc700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c2c098780 0x7f2c2c09ac30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c400a28f0 tx=0x7f2c30007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.069+0000 7f2c26ffd700 1 -- 172.21.15.121:0/2283604923 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c3408fb40 con 0x7f2c400630e0 2023-09-29T23:06:28.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.253+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c40135f40 con 0x7f2c400630e0 2023-09-29T23:06:28.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.254+0000 7f2c26ffd700 1 -- 172.21.15.121:0/2283604923 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12254 (secure 0 0 0) 0x7f2c34093160 con 0x7f2c400630e0 2023-09-29T23:06:28.253 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:28.254 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":19,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:06:19.255324+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-29T23:06:16.382138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-29T23:06:17.218163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.255+0000 7f2c45724700 10 mgrc shutdown 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.255+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c2c098780 msgr2=0x7f2c2c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.255+0000 7f2c45724700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c2c098780 0x7f2c2c09ac30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c400a28f0 tx=0x7f2c30007d30 comp rx=0 tx=0).stop 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.255+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 msgr2=0x7f2c4009cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.255+0000 7f2c45724700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c4009cc10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c40064110 tx=0x7f2c34016000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 shutdown_connections 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c2c098780 0x7f2c2c09ac30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c40142540 0x7f2c40133fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c400630e0 0x7f2c4009cc10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 --2- 172.21.15.121:0/2283604923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400634b0 0x7f2c401360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 >> 172.21.15.121:0/2283604923 conn(0x7f2c400a5bc0 msgr2=0x7f2c400b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 shutdown_connections 2023-09-29T23:06:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.256+0000 7f2c45724700 1 -- 172.21.15.121:0/2283604923 wait complete. 2023-09-29T23:06:28.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-29T23:06:28.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-29T23:06:28.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-29T23:06:28.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-29T23:06:28.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.395+0000 7f9a4d598700 1 Processor -- start 2023-09-29T23:06:28.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.395+0000 7fe6e22b0700 1 Processor -- start 2023-09-29T23:06:28.396 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.395+0000 7f9a4d598700 1 -- start start 2023-09-29T23:06:28.396 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a4d598700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a480a5800 0x7f9a48069a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.396 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a4d598700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 0x7f9a48069fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a4d598700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a48067a60 con 0x7f9a48067690 2023-09-29T23:06:28.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a4d598700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4806dfc0 con 0x7f9a480a5800 2023-09-29T23:06:28.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a4d598700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4806a4e0 con 0x7f9a480a5c40 2023-09-29T23:06:28.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a467fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a480a5800 0x7f9a48069a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a467fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a480a5800 0x7f9a48069a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58556/0 (socket says 172.21.15.121:58556) 2023-09-29T23:06:28.397 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7f9a467fc700 1 -- 172.21.15.121:0/1286908075 learned_addr learned my addr 172.21.15.121:0/1286908075 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.395+0000 7fe6e22b0700 1 -- start start 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7fe6e22b0700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe6dc140a90 0x7fe6dc140e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7fe6e22b0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe6dc1413d0 0x7fe6dc147d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7fe6e22b0700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6dc09df40 con 0x7fe6dc0b39b0 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7fe6e22b0700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6dc09d6d0 con 0x7fe6dc1413d0 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7fe6e22b0700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6dc09e500 con 0x7fe6dc140a90 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.396+0000 7fe6e084d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe6dc1413d0 0x7fe6dc147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6e084d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe6dc1413d0 0x7fe6dc147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58572/0 (socket says 172.21.15.121:58572) 2023-09-29T23:06:28.398 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6e084d700 1 -- 172.21.15.121:0/2992822169 learned_addr learned my addr 172.21.15.121:0/2992822169 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6db7fe700 1 --2- 172.21.15.121:0/2992822169 >> v2:172.21.15.123:3300/0 conn(0x7fe6dc140a90 0x7fe6dc140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a467fc700 1 -- 172.21.15.121:0/1286908075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 msgr2=0x7f9a48069fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a467fc700 1 --2- 172.21.15.121:0/1286908075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 0x7f9a48069fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a467fc700 1 -- 172.21.15.121:0/1286908075 >> v1:172.21.15.123:6789/0 conn(0x7f9a48067690 legacy=0x7f9a4806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.399 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a467fc700 1 -- 172.21.15.121:0/1286908075 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4806e580 con 0x7f9a480a5800 2023-09-29T23:06:28.400 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6e084d700 1 -- 172.21.15.121:0/2992822169 >> v2:172.21.15.123:3300/0 conn(0x7fe6dc140a90 msgr2=0x7fe6dc140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.400 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6e084d700 1 --2- 172.21.15.121:0/2992822169 >> v2:172.21.15.123:3300/0 conn(0x7fe6dc140a90 0x7fe6dc140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.400 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6e084d700 1 -- 172.21.15.121:0/2992822169 >> v1:172.21.15.121:6789/0 conn(0x7fe6dc0b39b0 legacy=0x7fe6dc0a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.400 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6e084d700 1 -- 172.21.15.121:0/2992822169 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc09e7a0 con 0x7fe6dc1413d0 2023-09-29T23:06:28.400 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6e084d700 1 --2- 172.21.15.121:0/2992822169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc147d60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc09f000 tx=0x7fe6d400a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671a958b68374ed4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.400 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a467fc700 1 --2- 172.21.15.121:0/1286908075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480a5800 0x7f9a48069a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0099b0 tx=0x7f9a3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4f2b48c80bfd7d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.401 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6daffd700 1 -- 172.21.15.121:0/2992822169 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d400ca40 con 0x7fe6dc1413d0 2023-09-29T23:06:28.401 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6daffd700 1 -- 172.21.15.121:0/2992822169 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d400cba0 con 0x7fe6dc1413d0 2023-09-29T23:06:28.401 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7fe6daffd700 1 -- 172.21.15.121:0/2992822169 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4018400 con 0x7fe6dc1413d0 2023-09-29T23:06:28.402 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a45ffb700 1 -- 172.21.15.121:0/1286908075 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00b860 con 0x7f9a480a5800 2023-09-29T23:06:28.402 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a45ffb700 1 -- 172.21.15.121:0/1286908075 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c00b9c0 con 0x7f9a480a5800 2023-09-29T23:06:28.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.397+0000 7f9a45ffb700 1 -- 172.21.15.121:0/1286908075 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00bd60 con 0x7f9a480a5800 2023-09-29T23:06:28.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 -- 172.21.15.121:0/1286908075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480a5800 msgr2=0x7f9a48069a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 --2- 172.21.15.121:0/1286908075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480a5800 0x7f9a48069a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0099b0 tx=0x7f9a3c009250 comp rx=0 tx=0).stop 2023-09-29T23:06:28.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 -- 172.21.15.121:0/1286908075 shutdown_connections 2023-09-29T23:06:28.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 --2- 172.21.15.121:0/1286908075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 0x7f9a48069fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 --2- 172.21.15.121:0/1286908075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480a5800 0x7f9a48069a60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.403 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 -- 172.21.15.121:0/1286908075 >> 172.21.15.121:0/1286908075 conn(0x7f9a48068550 msgr2=0x7f9a48068950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 -- 172.21.15.121:0/2992822169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 msgr2=0x7fe6dc147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 --2- 172.21.15.121:0/2992822169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc147d60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc09f000 tx=0x7fe6d400a3f0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 -- 172.21.15.121:0/1286908075 shutdown_connections 2023-09-29T23:06:28.404 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7f9a4d598700 1 -- 172.21.15.121:0/1286908075 wait complete. 2023-09-29T23:06:28.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 -- 172.21.15.121:0/2992822169 shutdown_connections 2023-09-29T23:06:28.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 --2- 172.21.15.121:0/2992822169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc147d60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 --2- 172.21.15.121:0/2992822169 >> v2:172.21.15.123:3300/0 conn(0x7fe6dc140a90 0x7fe6dc140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.405 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 -- 172.21.15.121:0/2992822169 >> 172.21.15.121:0/2992822169 conn(0x7fe6dc069ec0 msgr2=0x7fe6dc0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.406 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 -- 172.21.15.121:0/2992822169 shutdown_connections 2023-09-29T23:06:28.406 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.398+0000 7fe6e22b0700 1 -- 172.21.15.121:0/2992822169 wait complete. 2023-09-29T23:06:28.406 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 Processor -- start 2023-09-29T23:06:28.406 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 Processor -- start 2023-09-29T23:06:28.406 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 -- start start 2023-09-29T23:06:28.407 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 -- start start 2023-09-29T23:06:28.407 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48067690 0x7f9a4809d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.407 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a480a5800 0x7f9a48095c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.407 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 0x7f9a4809dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.407 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4809f180 con 0x7f9a48067690 2023-09-29T23:06:28.407 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4809e9f0 con 0x7f9a480a5800 2023-09-29T23:06:28.408 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7f9a4d598700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480a1820 con 0x7f9a480a5c40 2023-09-29T23:06:28.408 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6dc0b39b0 0x7fe6dc098210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.408 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6dc140a90 0x7fe6dc151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.408 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc06e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.408 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc151bf0 con 0x7fe6dc1413d0 2023-09-29T23:06:28.408 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc151eb0 con 0x7fe6dc140a90 2023-09-29T23:06:28.409 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.400+0000 7fe6e22b0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc152170 con 0x7fe6dc0b39b0 2023-09-29T23:06:28.409 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48067690 0x7f9a4809d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.409 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48067690 0x7f9a4809d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58584/0 (socket says 172.21.15.121:58584) 2023-09-29T23:06:28.409 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 -- 172.21.15.121:0/4010937622 learned_addr learned my addr 172.21.15.121:0/4010937622 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.410 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc06e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.410 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc06e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58586/0 (socket says 172.21.15.121:58586) 2023-09-29T23:06:28.410 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 -- 172.21.15.121:0/456167017 learned_addr learned my addr 172.21.15.121:0/456167017 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.410 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6dbfff700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6dc0b39b0 0x7fe6dc098210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.410 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6db7fe700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6dc140a90 0x7fe6dc151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.410 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a477fe700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 0x7f9a4809dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.411 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 -- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6dc0b39b0 msgr2=0x7fe6dc098210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.411 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6dc0b39b0 0x7fe6dc098210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.411 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 -- 172.21.15.121:0/456167017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6dc140a90 msgr2=0x7fe6dc151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.411 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6dc140a90 0x7fe6dc151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.411 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 -- 172.21.15.121:0/456167017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d400a690 con 0x7fe6dc1413d0 2023-09-29T23:06:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 -- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 msgr2=0x7f9a4809dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a480a5c40 0x7f9a4809dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 -- 172.21.15.121:0/4010937622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a480a5800 msgr2=0x7f9a48095c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a480a5800 0x7f9a48095c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 -- 172.21.15.121:0/4010937622 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c0094f0 con 0x7f9a48067690 2023-09-29T23:06:28.412 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a46ffd700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48067690 0x7f9a4809d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c5c0 tx=0x7f9a3800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7fe6e084d700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc06e9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400a430 tx=0x7fe6d401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.401+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38011590 con 0x7f9a48067690 2023-09-29T23:06:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38011710 con 0x7f9a48067690 2023-09-29T23:06:28.413 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38011ad0 con 0x7f9a48067690 2023-09-29T23:06:28.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4010600 con 0x7fe6dc1413d0 2023-09-29T23:06:28.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d4010780 con 0x7fe6dc1413d0 2023-09-29T23:06:28.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6dc06ef00 con 0x7fe6dc1413d0 2023-09-29T23:06:28.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4010b80 con 0x7fe6dc1413d0 2023-09-29T23:06:28.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6dc098ae0 con 0x7fe6dc1413d0 2023-09-29T23:06:28.415 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4809ff00 con 0x7f9a48067690 2023-09-29T23:06:28.415 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4809e2a0 con 0x7f9a48067690 2023-09-29T23:06:28.415 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.402+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe6dc06ab80 con 0x7fe6dc1413d0 2023-09-29T23:06:28.415 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.403+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9a48071660 con 0x7f9a48067690 2023-09-29T23:06:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe6d402a3b0 con 0x7fe6dc1413d0 2023-09-29T23:06:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7fe6c3fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9a3801b3b0 con 0x7f9a48067690 2023-09-29T23:06:28.416 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7f9a2effd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7fe6c3fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7fe6c3fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7fe6c3fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.404+0000 7fe6c3fff700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6c8098780 0x7fe6c809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fe6d40c1b30 con 0x7fe6dc1413d0 2023-09-29T23:06:28.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7fe6dbfff700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6c8098780 0x7fe6c809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7fe6c3fff700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6800/113368,v1:172.21.15.121:6801/113368] conn(0x7fe6c80a02e0 0x7fe6c80a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 --> [v2:172.21.15.121:6800/113368,v1:172.21.15.121:6801/113368] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6c80a2de0 con 0x7fe6c80a02e0 2023-09-29T23:06:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d40c6220 con 0x7fe6dc1413d0 2023-09-29T23:06:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9a34098780 0x7f9a3409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.419 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7fe6db7fe700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6800/113368,v1:172.21.15.121:6801/113368] conn(0x7fe6c80a02e0 0x7fe6c80a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.419 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a467fc700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9a34098780 0x7f9a3409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.419 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f9a380c00c0 con 0x7f9a48067690 2023-09-29T23:06:28.420 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.123:6800/115865,v1:172.21.15.123:6801/115865] conn(0x7f9a340a02e0 0x7f9a340a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.420 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 --> [v2:172.21.15.123:6800/115865,v1:172.21.15.123:6801/115865] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a340a2de0 con 0x7f9a340a02e0 2023-09-29T23:06:28.420 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a380c0490 con 0x7f9a48067690 2023-09-29T23:06:28.420 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a467fc700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9a34098780 0x7f9a3409ac30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0099b0 tx=0x7f9a3c017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.420 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.405+0000 7f9a477fe700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.123:6800/115865,v1:172.21.15.123:6801/115865] conn(0x7f9a340a02e0 0x7f9a340a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.420 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.406+0000 7fe6dbfff700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6c8098780 0x7fe6c809ac30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007fd0 tx=0x7fe6cc00e750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.406+0000 7f9a477fe700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.123:6800/115865,v1:172.21.15.123:6801/115865] conn(0x7f9a340a02e0 0x7f9a340a2710 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.406+0000 7fe6db7fe700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6800/113368,v1:172.21.15.121:6801/113368] conn(0x7fe6c80a02e0 0x7fe6c80a2710 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.407+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== osd.2 v2:172.21.15.123:6800/115865 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9a340a2de0 con 0x7f9a340a02e0 2023-09-29T23:06:28.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.407+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== osd.0 v2:172.21.15.121:6800/113368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe6c80a2de0 con 0x7fe6c80a02e0 2023-09-29T23:06:28.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.408+0000 7f3511ba1700 1 Processor -- start 2023-09-29T23:06:28.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.408+0000 7f3511ba1700 1 -- start start 2023-09-29T23:06:28.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.408+0000 7f40599f4700 1 Processor -- start 2023-09-29T23:06:28.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f3511ba1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f350c13da80 0x7f350c06e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f3511ba1700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f350c13dec0 0x7f350c147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f3511ba1700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c06a0d0 con 0x7f350c069d00 2023-09-29T23:06:28.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f3511ba1700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c0a5bc0 con 0x7f350c13da80 2023-09-29T23:06:28.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f3511ba1700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c06f1f0 con 0x7f350c13dec0 2023-09-29T23:06:28.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f40599f4700 1 -- start start 2023-09-29T23:06:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f350bfff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f350c13dec0 0x7f350c147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f350bfff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f350c13dec0 0x7f350c147ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.121:51902/0 (socket says 172.21.15.121:51902) 2023-09-29T23:06:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f350bfff700 1 -- 172.21.15.121:0/3131255267 learned_addr learned my addr 172.21.15.121:0/3131255267 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f350bfff700 1 -- 172.21.15.121:0/3131255267 >> v2:172.21.15.121:3300/0 conn(0x7f350c13da80 msgr2=0x7f350c06e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f350bfff700 1 --2- 172.21.15.121:0/3131255267 >> v2:172.21.15.121:3300/0 conn(0x7f350c13da80 0x7f350c06e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f350bfff700 1 -- 172.21.15.121:0/3131255267 >> v1:172.21.15.121:6789/0 conn(0x7f350c069d00 legacy=0x7f350c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.409+0000 7f350bfff700 1 -- 172.21.15.121:0/3131255267 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c13d2d0 con 0x7f350c13dec0 2023-09-29T23:06:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f350bfff700 1 --2- 172.21.15.121:0/3131255267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 0x7f350c147ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f350c06e770 tx=0x7f3504009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61339f1303546071 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40599f4700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f405409f820 0x7f40540a2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40599f4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40540a3cd0 0x7f40540a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40599f4700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4054063700 con 0x7f4054063330 2023-09-29T23:06:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40599f4700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40540a57e0 con 0x7f405409f820 2023-09-29T23:06:28.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40599f4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40540a0880 con 0x7f40540a3cd0 2023-09-29T23:06:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f350a7fc700 1 -- 172.21.15.121:0/3131255267 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350400e070 con 0x7f350c13dec0 2023-09-29T23:06:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f350a7fc700 1 -- 172.21.15.121:0/3131255267 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350400ba10 con 0x7f350c13dec0 2023-09-29T23:06:28.427 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f350a7fc700 1 -- 172.21.15.121:0/3131255267 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35040092e0 con 0x7f350c13dec0 2023-09-29T23:06:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f4052ffd700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4054063330 0x7f40540a35c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.121:51486/0 (socket says 172.21.15.121:51486) 2023-09-29T23:06:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f4052ffd700 1 -- 172.21.15.121:0/734436982 learned_addr learned my addr 172.21.15.121:0/734436982 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40537fe700 1 --2- 172.21.15.121:0/734436982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40540a3cd0 0x7f40540a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f4051ffb700 1 -- 172.21.15.121:0/734436982 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060857543 0 0) 0x7f4054063700 con 0x7f4054063330 2023-09-29T23:06:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f4051ffb700 1 -- 172.21.15.121:0/734436982 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c003570 con 0x7f4054063330 2023-09-29T23:06:28.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40537fe700 1 -- 172.21.15.121:0/734436982 >> v2:172.21.15.121:3300/0 conn(0x7f405409f820 msgr2=0x7f40540a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40537fe700 1 --2- 172.21.15.121:0/734436982 >> v2:172.21.15.121:3300/0 conn(0x7f405409f820 0x7f40540a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40537fe700 1 -- 172.21.15.121:0/734436982 >> v1:172.21.15.123:6789/0 conn(0x7f4054063330 legacy=0x7f40540a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40537fe700 1 -- 172.21.15.121:0/734436982 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40540a4250 con 0x7f40540a3cd0 2023-09-29T23:06:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.410+0000 7f40537fe700 1 --2- 172.21.15.121:0/734436982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40540a3cd0 0x7f40540a28a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f404c009e60 tx=0x7f404c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378ca63b87b93bfa server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f4051ffb700 1 -- 172.21.15.121:0/734436982 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00e070 con 0x7f40540a3cd0 2023-09-29T23:06:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 -- 172.21.15.121:0/3131255267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 msgr2=0x7f350c147ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 --2- 172.21.15.121:0/3131255267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 0x7f350c147ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f350c06e770 tx=0x7f3504009840 comp rx=0 tx=0).stop 2023-09-29T23:06:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f4051ffb700 1 -- 172.21.15.121:0/734436982 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c00bb60 con 0x7f40540a3cd0 2023-09-29T23:06:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f4051ffb700 1 -- 172.21.15.121:0/734436982 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c012430 con 0x7f40540a3cd0 2023-09-29T23:06:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 -- 172.21.15.121:0/3131255267 shutdown_connections 2023-09-29T23:06:28.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 --2- 172.21.15.121:0/3131255267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 0x7f350c147ac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 --2- 172.21.15.121:0/3131255267 >> v2:172.21.15.121:3300/0 conn(0x7f350c13da80 0x7f350c06e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 -- 172.21.15.121:0/3131255267 >> 172.21.15.121:0/3131255267 conn(0x7f350c068550 msgr2=0x7f350c068950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 -- 172.21.15.121:0/3131255267 shutdown_connections 2023-09-29T23:06:28.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.411+0000 7f3511ba1700 1 -- 172.21.15.121:0/3131255267 wait complete. 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 -- 172.21.15.121:0/734436982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40540a3cd0 msgr2=0x7f40540a28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 --2- 172.21.15.121:0/734436982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40540a3cd0 0x7f40540a28a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f404c009e60 tx=0x7f404c009780 comp rx=0 tx=0).stop 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 -- 172.21.15.121:0/734436982 shutdown_connections 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 --2- 172.21.15.121:0/734436982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40540a3cd0 0x7f40540a28a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 --2- 172.21.15.121:0/734436982 >> v2:172.21.15.121:3300/0 conn(0x7f405409f820 0x7f40540a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 -- 172.21.15.121:0/734436982 >> 172.21.15.121:0/734436982 conn(0x7f40540a5bc0 msgr2=0x7f40540b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 -- 172.21.15.121:0/734436982 shutdown_connections 2023-09-29T23:06:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f40599f4700 1 -- 172.21.15.121:0/734436982 wait complete. 2023-09-29T23:06:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.412+0000 7f3511ba1700 1 Processor -- start 2023-09-29T23:06:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f3511ba1700 1 -- start start 2023-09-29T23:06:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f3511ba1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c069d00 0x7f350c09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f3511ba1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f350c13da80 0x7f350c140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f40599f4700 1 Processor -- start 2023-09-29T23:06:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f3511ba1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 0x7f350c150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f3511ba1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c09e3b0 con 0x7f350c069d00 2023-09-29T23:06:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f3511ba1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c151010 con 0x7f350c13dec0 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.413+0000 7f3511ba1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1512d0 con 0x7f350c13da80 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c069d00 0x7f350c09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c069d00 0x7f350c09f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58600/0 (socket says 172.21.15.121:58600) 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350b7fe700 1 -- 172.21.15.121:0/1187321003 learned_addr learned my addr 172.21.15.121:0/1187321003 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350affd700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f350c13da80 0x7f350c140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350bfff700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 0x7f350c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350bfff700 1 -- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f350c13da80 msgr2=0x7f350c140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350bfff700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f350c13da80 0x7f350c140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350bfff700 1 -- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c069d00 msgr2=0x7f350c09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350bfff700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c069d00 0x7f350c09f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350bfff700 1 -- 172.21.15.121:0/1187321003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504009ae0 con 0x7f350c13dec0 2023-09-29T23:06:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f350bfff700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 0x7f350c150930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3504014000 tx=0x7f3504017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f40599f4700 1 -- start start 2023-09-29T23:06:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35040092e0 con 0x7f350c13dec0 2023-09-29T23:06:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f40599f4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054063330 0x7f405409e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.414+0000 7f40599f4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405409f820 0x7f4054157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40599f4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40540a3cd0 0x7f4054158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40599f4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40541585c0 con 0x7f4054063330 2023-09-29T23:06:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40599f4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405406ec70 con 0x7f40540a3cd0 2023-09-29T23:06:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40599f4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40541587d0 con 0x7f405409f820 2023-09-29T23:06:28.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3504012d00 con 0x7f350c13dec0 2023-09-29T23:06:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350401e660 con 0x7f350c13dec0 2023-09-29T23:06:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f4052ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054063330 0x7f405409e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f4052ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054063330 0x7f405409e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58608/0 (socket says 172.21.15.121:58608) 2023-09-29T23:06:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40527fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405409f820 0x7f4054157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f4052ffd700 1 -- 172.21.15.121:0/3421346546 learned_addr learned my addr 172.21.15.121:0/3421346546 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40537fe700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40540a3cd0 0x7f4054158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c0a18b0 con 0x7f350c13dec0 2023-09-29T23:06:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c0a1aa0 con 0x7f350c13dec0 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40537fe700 1 -- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405409f820 msgr2=0x7f4054157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40537fe700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f405409f820 0x7f4054157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40537fe700 1 -- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054063330 msgr2=0x7f405409e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40537fe700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054063330 0x7f405409e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40537fe700 1 -- 172.21.15.121:0/3421346546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c009a20 con 0x7f40540a3cd0 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.415+0000 7f40537fe700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40540a3cd0 0x7f4054158080 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f404c016000 tx=0x7f404c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.416+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c012cd0 con 0x7f40540a3cd0 2023-09-29T23:06:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.416+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c021070 con 0x7f40540a3cd0 2023-09-29T23:06:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.416+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c026680 con 0x7f40540a3cd0 2023-09-29T23:06:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.416+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405406de50 con 0x7f40540a3cd0 2023-09-29T23:06:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.416+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4054098740 con 0x7f40540a3cd0 2023-09-29T23:06:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.416+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f350c09dda0 con 0x7f350c13dec0 2023-09-29T23:06:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.418+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f350401ea90 con 0x7f350c13dec0 2023-09-29T23:06:28.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.418+0000 7f34f37fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.417+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f405406abd0 con 0x7f40540a3cd0 2023-09-29T23:06:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.418+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f404c023030 con 0x7f40540a3cd0 2023-09-29T23:06:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.418+0000 7f403affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4040098700 0x7f404009abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f34f80987d0 0x7f34f809ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f4052ffd700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4040098700 0x7f404009abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f404c0c2ae0 con 0x7f40540a3cd0 2023-09-29T23:06:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:6816/115891,v1:172.21.15.123:6817/115891] conn(0x7f40400a0260 0x7f40400a2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 --> [v2:172.21.15.123:6816/115891,v1:172.21.15.123:6817/115891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40400a2d60 con 0x7f40400a0260 2023-09-29T23:06:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f404c0c2e80 con 0x7f40540a3cd0 2023-09-29T23:06:28.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f35040c2980 con 0x7f350c13dec0 2023-09-29T23:06:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:6808/115866,v1:172.21.15.123:6809/115866] conn(0x7f34f80a0330 0x7f34f80a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f350b7fe700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f34f80987d0 0x7f34f809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 --> [v2:172.21.15.123:6808/115866,v1:172.21.15.123:6809/115866] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34f80a2e30 con 0x7f34f80a0330 2023-09-29T23:06:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35040c2d50 con 0x7f350c13dec0 2023-09-29T23:06:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f40527fc700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:6816/115891,v1:172.21.15.123:6817/115891] conn(0x7f40400a0260 0x7f40400a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f350affd700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:6808/115866,v1:172.21.15.123:6809/115866] conn(0x7f34f80a0330 0x7f34f80a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.419+0000 7f350b7fe700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f34f80987d0 0x7f34f809ac80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00be30 tx=0x7f34fc00b430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.420+0000 7f4052ffd700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4040098700 0x7f404009abb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4044000c00 tx=0x7f4044009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.420+0000 7f40527fc700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:6816/115891,v1:172.21.15.123:6817/115891] conn(0x7f40400a0260 0x7f40400a2690 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.420+0000 7f350affd700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:6808/115866,v1:172.21.15.123:6809/115866] conn(0x7f34f80a0330 0x7f34f80a2760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.420+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== osd.3 v2:172.21.15.123:6808/115866 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f34f80a2e30 con 0x7f34f80a0330 2023-09-29T23:06:28.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.420+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== osd.1 v2:172.21.15.123:6816/115891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f40400a2d60 con 0x7f40400a0260 2023-09-29T23:06:28.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.426+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 --> [v2:172.21.15.121:6800/113368,v1:172.21.15.121:6801/113368] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe6dc071660 con 0x7fe6c80a02e0 2023-09-29T23:06:28.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.427+0000 7fe6c3fff700 1 -- 172.21.15.121:0/456167017 <== osd.0 v2:172.21.15.121:6800/113368 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe6dc071660 con 0x7fe6c80a02e0 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.427+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6800/113368,v1:172.21.15.121:6801/113368] conn(0x7fe6c80a02e0 msgr2=0x7fe6c80a2710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.427+0000 7fe6e22b0700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6800/113368,v1:172.21.15.121:6801/113368] conn(0x7fe6c80a02e0 0x7fe6c80a2710 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.427+0000 7fe6e22b0700 10 mgrc shutdown 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6c8098780 msgr2=0x7fe6c809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe6c8098780 0x7fe6c809ac30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007fd0 tx=0x7fe6cc00e750 comp rx=0 tx=0).stop 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 msgr2=0x7fe6dc06e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 --2- 172.21.15.121:0/456167017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6dc1413d0 0x7fe6dc06e9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400a430 tx=0x7fe6d401a000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e084d700 1 -- 172.21.15.121:0/456167017 reap_dead start 2023-09-29T23:06:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 shutdown_connections 2023-09-29T23:06:28.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 >> 172.21.15.121:0/456167017 conn(0x7fe6dc069ec0 msgr2=0x7fe6dc0a59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 shutdown_connections 2023-09-29T23:06:28.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.429+0000 7fe6e22b0700 1 -- 172.21.15.121:0/456167017 wait complete. 2023-09-29T23:06:28.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.432+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 --> [v2:172.21.15.123:6800/115865,v1:172.21.15.123:6801/115865] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9a480b2400 con 0x7f9a340a02e0 2023-09-29T23:06:28.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.433+0000 7f9a2effd700 1 -- 172.21.15.121:0/4010937622 <== osd.2 v2:172.21.15.123:6800/115865 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9a480b2400 con 0x7f9a340a02e0 2023-09-29T23:06:28.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.433+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 >> [v2:172.21.15.123:6800/115865,v1:172.21.15.123:6801/115865] conn(0x7f9a340a02e0 msgr2=0x7f9a340a2710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.433+0000 7f9a4d598700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.123:6800/115865,v1:172.21.15.123:6801/115865] conn(0x7f9a340a02e0 0x7f9a340a2710 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.434+0000 7f9a4d598700 10 mgrc shutdown 2023-09-29T23:06:28.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.434+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9a34098780 msgr2=0x7f9a3409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.434+0000 7f9a4d598700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9a34098780 0x7f9a3409ac30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0099b0 tx=0x7f9a3c017000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.434+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48067690 msgr2=0x7f9a4809d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.434+0000 7f9a4d598700 1 --2- 172.21.15.121:0/4010937622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48067690 0x7f9a4809d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c5c0 tx=0x7f9a3800c8d0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.434+0000 7f9a477fe700 1 -- 172.21.15.121:0/4010937622 reap_dead start 2023-09-29T23:06:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.436+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 --> [v2:172.21.15.123:6816/115891,v1:172.21.15.123:6817/115891] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4054071660 con 0x7f40400a0260 2023-09-29T23:06:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 shutdown_connections 2023-09-29T23:06:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 >> 172.21.15.121:0/4010937622 conn(0x7f9a48068550 msgr2=0x7f9a480b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 shutdown_connections 2023-09-29T23:06:28.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f9a4d598700 1 -- 172.21.15.121:0/4010937622 wait complete. 2023-09-29T23:06:28.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f403affd700 1 -- 172.21.15.121:0/3421346546 <== osd.1 v2:172.21.15.123:6816/115891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4054071660 con 0x7f40400a0260 2023-09-29T23:06:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:6816/115891,v1:172.21.15.123:6817/115891] conn(0x7f40400a0260 msgr2=0x7f40400a2690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f40599f4700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:6816/115891,v1:172.21.15.123:6817/115891] conn(0x7f40400a0260 0x7f40400a2690 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f40599f4700 10 mgrc shutdown 2023-09-29T23:06:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4040098700 msgr2=0x7f404009abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.437+0000 7f40599f4700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4040098700 0x7f404009abb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4044000c00 tx=0x7f4044009040 comp rx=0 tx=0).stop 2023-09-29T23:06:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.438+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40540a3cd0 msgr2=0x7f4054158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.438+0000 7f40599f4700 1 --2- 172.21.15.121:0/3421346546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40540a3cd0 0x7f4054158080 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f404c016000 tx=0x7f404c019000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.438+0000 7f40537fe700 1 -- 172.21.15.121:0/3421346546 reap_dead start 2023-09-29T23:06:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.438+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 shutdown_connections 2023-09-29T23:06:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.438+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 >> 172.21.15.121:0/3421346546 conn(0x7f40540a5bc0 msgr2=0x7f40541446b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.438+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 shutdown_connections 2023-09-29T23:06:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.438+0000 7f40599f4700 1 -- 172.21.15.121:0/3421346546 wait complete. 2023-09-29T23:06:28.456 INFO:teuthology.orchestra.run.smithi121.stdout:38654705674 2023-09-29T23:06:28.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-29T23:06:28.457 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.443+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 --> [v2:172.21.15.123:6808/115866,v1:172.21.15.123:6809/115866] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f350c071660 con 0x7f34f80a0330 2023-09-29T23:06:28.457 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.443+0000 7f34f37fe700 1 -- 172.21.15.121:0/1187321003 <== osd.3 v2:172.21.15.123:6808/115866 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f350c071660 con 0x7f34f80a0330 2023-09-29T23:06:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:6808/115866,v1:172.21.15.123:6809/115866] conn(0x7f34f80a0330 msgr2=0x7f34f80a2760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:6808/115866,v1:172.21.15.123:6809/115866] conn(0x7f34f80a0330 0x7f34f80a2760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 10 mgrc shutdown 2023-09-29T23:06:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f34f80987d0 msgr2=0x7f34f809ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f34f80987d0 0x7f34f809ac80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00be30 tx=0x7f34fc00b430 comp rx=0 tx=0).stop 2023-09-29T23:06:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 msgr2=0x7f350c150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 --2- 172.21.15.121:0/1187321003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f350c13dec0 0x7f350c150930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3504014000 tx=0x7f3504017000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f350bfff700 1 -- 172.21.15.121:0/1187321003 reap_dead start 2023-09-29T23:06:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 shutdown_connections 2023-09-29T23:06:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 >> 172.21.15.121:0/1187321003 conn(0x7f350c068550 msgr2=0x7f350c0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 shutdown_connections 2023-09-29T23:06:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.444+0000 7f3511ba1700 1 -- 172.21.15.121:0/1187321003 wait complete. 2023-09-29T23:06:28.459 INFO:teuthology.orchestra.run.smithi121.stdout:38654705672 2023-09-29T23:06:28.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-29T23:06:28.460 INFO:teuthology.orchestra.run.smithi121.stdout:38654705674 2023-09-29T23:06:28.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-29T23:06:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:38654705674 2023-09-29T23:06:28.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-29T23:06:28.625 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.626+0000 7fdf59913700 1 Processor -- start 2023-09-29T23:06:28.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.626+0000 7fb9060ec700 1 Processor -- start 2023-09-29T23:06:28.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.626+0000 7fdf59913700 1 -- start start 2023-09-29T23:06:28.627 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.626+0000 7fb9060ec700 1 -- start start 2023-09-29T23:06:28.627 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf59913700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdf540a4590 0x7fdf540af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.627 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf59913700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540b0390 con 0x7fdf5409c740 2023-09-29T23:06:28.627 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf59913700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540b0630 con 0x7fdf540632a0 2023-09-29T23:06:28.627 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf59913700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf54063670 con 0x7fdf540a4590 2023-09-29T23:06:28.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf527fc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdf5409c740 0x7fdf540af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44470/0 (socket says 172.21.15.121:44470) 2023-09-29T23:06:28.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf527fc700 1 -- 172.21.15.121:0/3730082059 learned_addr learned my addr 172.21.15.121:0/3730082059 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:28.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fb9060ec700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9000b3c00 0x7fb9000a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fb9060ec700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90009dca0 con 0x7fb9000a3c70 2023-09-29T23:06:28.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fb9060ec700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9000a3740 con 0x7fb9000b3830 2023-09-29T23:06:28.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fb9060ec700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90009eed0 con 0x7fb9000b3c00 2023-09-29T23:06:28.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf537fe700 1 --2- 172.21.15.121:0/3730082059 >> v2:172.21.15.123:3300/0 conn(0x7fdf540a4590 0x7fdf540af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fb8ff7fe700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fb9000b3830 0x7fb90009e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.121:51496/0 (socket says 172.21.15.121:51496) 2023-09-29T23:06:28.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fb8ff7fe700 1 -- 172.21.15.121:0/3340186329 learned_addr learned my addr 172.21.15.121:0/3340186329 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:28.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf51ffb700 1 -- 172.21.15.121:0/3730082059 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305419327 0 0) 0x7fdf540b0390 con 0x7fdf5409c740 2023-09-29T23:06:28.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf51ffb700 1 -- 172.21.15.121:0/3730082059 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c003570 con 0x7fdf5409c740 2023-09-29T23:06:28.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf537fe700 1 -- 172.21.15.121:0/3730082059 >> v1:172.21.15.123:6789/0 conn(0x7fdf540632a0 legacy=0x7fdf540a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf537fe700 1 -- 172.21.15.121:0/3730082059 >> v1:172.21.15.121:6789/0 conn(0x7fdf5409c740 legacy=0x7fdf540af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.627+0000 7fdf537fe700 1 -- 172.21.15.121:0/3730082059 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540a4960 con 0x7fdf540a4590 2023-09-29T23:06:28.631 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283819239 0 0) 0x7fb9000a3740 con 0x7fb9000b3830 2023-09-29T23:06:28.631 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ec004680 con 0x7fb9000b3830 2023-09-29T23:06:28.631 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107277893 0 0) 0x7fb90009dca0 con 0x7fb9000a3c70 2023-09-29T23:06:28.631 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9000a3740 con 0x7fb9000a3c70 2023-09-29T23:06:28.631 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fdf537fe700 1 --2- 172.21.15.121:0/3730082059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540a4590 0x7fdf540af910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009e60 tx=0x7fdf4c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2bd73a21cf31df4 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251211881 0 0) 0x7fb8ec004680 con 0x7fb9000b3830 2023-09-29T23:06:28.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 >> v2:172.21.15.123:3300/0 conn(0x7fb9000b3c00 msgr2=0x7fb9000a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 --2- 172.21.15.121:0/3340186329 >> v2:172.21.15.123:3300/0 conn(0x7fb9000b3c00 0x7fb9000a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 >> v1:172.21.15.121:6789/0 conn(0x7fb9000a3c70 legacy=0x7fb9001474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9000a57c0 con 0x7fb9000b3830 2023-09-29T23:06:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7fb8e80014c0 con 0x7fb9000b3830 2023-09-29T23:06:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fdf51ffb700 1 -- 172.21.15.121:0/3730082059 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c00e070 con 0x7fdf540a4590 2023-09-29T23:06:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 >> v1:172.21.15.123:6789/0 conn(0x7fb9000b3830 legacy=0x7fb90009e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec004d00 0x7fb8ec0050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fdf51ffb700 1 -- 172.21.15.121:0/3730082059 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c00bbf0 con 0x7fdf540a4590 2023-09-29T23:06:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fdf51ffb700 1 -- 172.21.15.121:0/3730082059 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c012430 con 0x7fdf540a4590 2023-09-29T23:06:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8ec005920 0x7fb8ec005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec0063f0 0x7fb8ec00cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f80015b0 con 0x7fb8ec004d00 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec00d370 con 0x7fb8ec0063f0 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec00d530 con 0x7fb8ec005920 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec004d00 0x7fb8ec0050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fffff700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec0063f0 0x7fb8ec00cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8ff7fe700 1 -- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8ec005920 msgr2=0x7fb8ec005d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8ec005920 0x7fb8ec005d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8ff7fe700 1 -- 172.21.15.121:0/3340186329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec0063f0 msgr2=0x7fb8ec00cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec0063f0 0x7fb8ec00cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8ff7fe700 1 -- 172.21.15.121:0/3340186329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9000a57c0 con 0x7fb8ec004d00 2023-09-29T23:06:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.628+0000 7fb8fffff700 1 -- 172.21.15.121:0/3340186329 reap_dead start 2023-09-29T23:06:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec004d00 0x7fb8ec0050d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800c360 tx=0x7fb8e8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef47c46be445d2cc server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800cc10 con 0x7fb8ec004d00 2023-09-29T23:06:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e800cd70 con 0x7fb8ec004d00 2023-09-29T23:06:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fb8fe7fc700 1 -- 172.21.15.121:0/3340186329 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800cee0 con 0x7fb8ec004d00 2023-09-29T23:06:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fdf59913700 1 -- 172.21.15.121:0/3730082059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540a4590 msgr2=0x7fdf540af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fdf59913700 1 --2- 172.21.15.121:0/3730082059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540a4590 0x7fdf540af910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009e60 tx=0x7fdf4c009780 comp rx=0 tx=0).stop 2023-09-29T23:06:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fdf59913700 1 -- 172.21.15.121:0/3730082059 shutdown_connections 2023-09-29T23:06:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fdf59913700 1 --2- 172.21.15.121:0/3730082059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540a4590 0x7fdf540af910 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fdf59913700 1 -- 172.21.15.121:0/3730082059 >> 172.21.15.121:0/3730082059 conn(0x7fdf54069ec0 msgr2=0x7fdf540b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fdf59913700 1 -- 172.21.15.121:0/3730082059 shutdown_connections 2023-09-29T23:06:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fdf59913700 1 -- 172.21.15.121:0/3730082059 wait complete. 2023-09-29T23:06:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fb9060ec700 1 -- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec004d00 msgr2=0x7fb8ec0050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.629+0000 7fb9060ec700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec004d00 0x7fb8ec0050d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800c360 tx=0x7fb8e8001e70 comp rx=0 tx=0).stop 2023-09-29T23:06:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.630+0000 7fb9060ec700 1 -- 172.21.15.121:0/3340186329 shutdown_connections 2023-09-29T23:06:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.630+0000 7fb9060ec700 1 --2- 172.21.15.121:0/3340186329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec004d00 0x7fb8ec0050d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.630+0000 7fb9060ec700 1 -- 172.21.15.121:0/3340186329 >> 172.21.15.121:0/3340186329 conn(0x7fb9000a5bc0 msgr2=0x7fb9000b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.630+0000 7fb9060ec700 1 -- 172.21.15.121:0/3340186329 shutdown_connections 2023-09-29T23:06:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.630+0000 7fb9060ec700 1 -- 172.21.15.121:0/3340186329 wait complete. 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.630+0000 7fdf59913700 1 Processor -- start 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.630+0000 7fdf59913700 1 -- start start 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf59913700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540632a0 0x7fdf540a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf59913700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf5409c740 0x7fdf54144b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf59913700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf540a4590 0x7fdf54145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf59913700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5406de50 con 0x7fdf5409c740 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf59913700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540a2f80 con 0x7fdf540632a0 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf59913700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5409b660 con 0x7fdf540a4590 2023-09-29T23:06:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf527fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf5409c740 0x7fdf54144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.642 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf527fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf5409c740 0x7fdf54144b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58618/0 (socket says 172.21.15.121:58618) 2023-09-29T23:06:28.642 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf527fc700 1 -- 172.21.15.121:0/4112265372 learned_addr learned my addr 172.21.15.121:0/4112265372 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.642 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fdf52ffd700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540632a0 0x7fdf540a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.642 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fb9060ec700 1 Processor -- start 2023-09-29T23:06:28.642 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.631+0000 7fb9060ec700 1 -- start start 2023-09-29T23:06:28.643 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf52ffd700 1 -- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf540a4590 msgr2=0x7fdf54145060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.643 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf52ffd700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf540a4590 0x7fdf54145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.645 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf52ffd700 1 -- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf5409c740 msgr2=0x7fdf54144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.645 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf52ffd700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf5409c740 0x7fdf54144b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.645 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf52ffd700 1 -- 172.21.15.121:0/4112265372 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c009a20 con 0x7fdf540632a0 2023-09-29T23:06:28.645 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf527fc700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf5409c740 0x7fdf54144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:28.646 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf52ffd700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540632a0 0x7fdf540a2460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400c490 tx=0x7fdf44007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.646 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf3affd700 1 -- 172.21.15.121:0/4112265372 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44011590 con 0x7fdf540632a0 2023-09-29T23:06:28.646 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf3affd700 1 -- 172.21.15.121:0/4112265372 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf44011710 con 0x7fdf540632a0 2023-09-29T23:06:28.646 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf3affd700 1 -- 172.21.15.121:0/4112265372 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44011ad0 con 0x7fdf540632a0 2023-09-29T23:06:28.647 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fb9060ec700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec004d00 0x7fb900145c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fb9060ec700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000a2b60 0x7fb900099e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fb9060ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90009b2f0 0x7fb900140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fb9060ec700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90009aff0 con 0x7fb90009b2f0 2023-09-29T23:06:28.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fb9060ec700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90009c6d0 con 0x7fb8ec004d00 2023-09-29T23:06:28.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fb9060ec700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90009bc90 con 0x7fb9000a2b60 2023-09-29T23:06:28.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fb8fffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90009b2f0 0x7fb900140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90009b2f0 0x7fb900140090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58624/0 (socket says 172.21.15.121:58624) 2023-09-29T23:06:28.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 -- 172.21.15.121:0/484912769 learned_addr learned my addr 172.21.15.121:0/484912769 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec004d00 0x7fb900145c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.632+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf540a0d10 con 0x7fdf540632a0 2023-09-29T23:06:28.650 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5406e540 con 0x7fdf540632a0 2023-09-29T23:06:28.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 -- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000a2b60 msgr2=0x7fb900099e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-29T23:06:28.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000a2b60 0x7fb900099e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 -- 172.21.15.121:0/484912769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec004d00 msgr2=0x7fb900145c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec004d00 0x7fb900145c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 -- 172.21.15.121:0/484912769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90009a470 con 0x7fb90009b2f0 2023-09-29T23:06:28.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec004d00 0x7fb900145c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:06:28.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8fffff700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90009b2f0 0x7fb900140090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb90006bc40 tx=0x7fb8f8001cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8f77fe700 1 -- 172.21.15.121:0/484912769 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8001080 con 0x7fb90009b2f0 2023-09-29T23:06:28.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8f77fe700 1 -- 172.21.15.121:0/484912769 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f800cae0 con 0x7fb90009b2f0 2023-09-29T23:06:28.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb8f77fe700 1 -- 172.21.15.121:0/484912769 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f800ce60 con 0x7fb90009b2f0 2023-09-29T23:06:28.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90009a730 con 0x7fb90009b2f0 2023-09-29T23:06:28.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.633+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90009b770 con 0x7fb90009b2f0 2023-09-29T23:06:28.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.634+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5406ab80 con 0x7fdf540632a0 2023-09-29T23:06:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.635+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90006abd0 con 0x7fb90009b2f0 2023-09-29T23:06:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fe0307ab700 1 Processor -- start 2023-09-29T23:06:28.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fe0307ab700 1 -- start start 2023-09-29T23:06:28.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8f77fe700 1 -- 172.21.15.121:0/484912769 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb8f8013630 con 0x7fb90009b2f0 2023-09-29T23:06:28.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8f77fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8f77fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8f77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8f77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8f77fe700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8e0098780 0x7fb8e009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8e0098780 0x7fb8e009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.636+0000 7fb8f77fe700 1 -- 172.21.15.121:0/484912769 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fb8f80c0780 con 0x7fb90009b2f0 2023-09-29T23:06:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd1a579a700 1 Processor -- start 2023-09-29T23:06:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd1a579a700 1 -- start start 2023-09-29T23:06:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fe0307ab700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe0280b4ba0 0x7fe0281444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fe0307ab700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe028143d00 con 0x7fe02806f960 2023-09-29T23:06:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fe0307ab700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe028143fc0 con 0x7fe028141340 2023-09-29T23:06:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fe0307ab700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0280b4f70 con 0x7fe0280b4ba0 2023-09-29T23:06:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fb8ff7fe700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8e0098780 0x7fb8e009ac30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000c00 tx=0x7fb8e8015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.658 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fe02ed48700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe028141340 0x7fe02814af00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.121:51504/0 (socket says 172.21.15.121:51504) 2023-09-29T23:06:28.658 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fe02ed48700 1 -- 172.21.15.121:0/4155497734 learned_addr learned my addr 172.21.15.121:0/4155497734 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd1a579a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a00a5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd1a579a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd1a009f2b0 0x7fd1a009f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fdf3affd700 1 -- 172.21.15.121:0/4112265372 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fdf4401b3b0 con 0x7fdf540632a0 2023-09-29T23:06:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fdf3affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fdf3affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fdf3affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fdf3affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fdf3affd700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdf40098780 0x7fdf4009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd1a579a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a0063700 con 0x7fd1a0144ad0 2023-09-29T23:06:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd1a579a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a009fbc0 con 0x7fd1a009f2b0 2023-09-29T23:06:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd1a579a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a0145610 con 0x7fd1a0063330 2023-09-29T23:06:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd19effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a00a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd19effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a00a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36664/0 (socket says 172.21.15.121:36664) 2023-09-29T23:06:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.637+0000 7fd19effd700 1 -- 172.21.15.121:0/863146403 learned_addr learned my addr 172.21.15.121:0/863146403 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fdf3affd700 1 -- 172.21.15.121:0/4112265372 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fdf440bfc60 con 0x7fdf540632a0 2023-09-29T23:06:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fe02e547700 1 --2- 172.21.15.121:0/4155497734 >> v2:172.21.15.121:3300/0 conn(0x7fe0280b4ba0 0x7fe0281444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19e7fc700 1 --2- 172.21.15.121:0/863146403 >> v2:172.21.15.123:3300/0 conn(0x7fd1a009f2b0 0x7fd1a009f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.663 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fdf527fc700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdf40098780 0x7fdf4009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.663 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19effd700 1 -- 172.21.15.121:0/863146403 >> v2:172.21.15.123:3300/0 conn(0x7fd1a009f2b0 msgr2=0x7fd1a009f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:28.663 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19effd700 1 --2- 172.21.15.121:0/863146403 >> v2:172.21.15.123:3300/0 conn(0x7fd1a009f2b0 0x7fd1a009f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19effd700 1 -- 172.21.15.121:0/863146403 >> v1:172.21.15.123:6789/0 conn(0x7fd1a0144ad0 legacy=0x7fd1a01480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19effd700 1 -- 172.21.15.121:0/863146403 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a01458d0 con 0x7fd1a0063330 2023-09-29T23:06:28.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19e7fc700 1 --2- 172.21.15.121:0/863146403 >> v2:172.21.15.123:3300/0 conn(0x7fd1a009f2b0 0x7fd1a009f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:06:28.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fe02e547700 1 -- 172.21.15.121:0/4155497734 >> v1:172.21.15.123:6789/0 conn(0x7fe028141340 legacy=0x7fe02814af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fe02e547700 1 -- 172.21.15.121:0/4155497734 >> v1:172.21.15.121:6789/0 conn(0x7fe02806f960 legacy=0x7fe028140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fe02e547700 1 -- 172.21.15.121:0/4155497734 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0281449e0 con 0x7fe0280b4ba0 2023-09-29T23:06:28.665 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19effd700 1 --2- 172.21.15.121:0/863146403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a00a5080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd194009990 tx=0x7fd194009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5393977d2f301d75 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.665 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fdf527fc700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdf40098780 0x7fdf4009ac30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf5409c2b0 tx=0x7fdf48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fe02e547700 1 --2- 172.21.15.121:0/4155497734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0280b4ba0 0x7fe0281444a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe0240098e0 tx=0x7fe0240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd76f95025fb124e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19dffb700 1 -- 172.21.15.121:0/863146403 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194005c10 con 0x7fd1a0063330 2023-09-29T23:06:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19dffb700 1 -- 172.21.15.121:0/863146403 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd194005d70 con 0x7fd1a0063330 2023-09-29T23:06:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.638+0000 7fd19dffb700 1 -- 172.21.15.121:0/863146403 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19400b880 con 0x7fd1a0063330 2023-09-29T23:06:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 -- 172.21.15.121:0/863146403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 msgr2=0x7fd1a00a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 --2- 172.21.15.121:0/863146403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a00a5080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd194009990 tx=0x7fd194009220 comp rx=0 tx=0).stop 2023-09-29T23:06:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 -- 172.21.15.121:0/863146403 shutdown_connections 2023-09-29T23:06:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 --2- 172.21.15.121:0/863146403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a00a5080 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 --2- 172.21.15.121:0/863146403 >> v2:172.21.15.123:3300/0 conn(0x7fd1a009f2b0 0x7fd1a009f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 -- 172.21.15.121:0/863146403 >> 172.21.15.121:0/863146403 conn(0x7fd1a00a5bc0 msgr2=0x7fd1a00b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 -- 172.21.15.121:0/863146403 shutdown_connections 2023-09-29T23:06:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.639+0000 7fd1a579a700 1 -- 172.21.15.121:0/863146403 wait complete. 2023-09-29T23:06:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.640+0000 7fdf3affd700 1 -- 172.21.15.121:0/4112265372 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdf4408d300 con 0x7fdf540632a0 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.640+0000 7fd1a579a700 1 Processor -- start 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd1a579a700 1 -- start start 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd1a579a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a006ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd1a579a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1a009f2b0 0x7fd1a006e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd1a579a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1a0144ad0 0x7fd1a0142450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd1a579a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a01589c0 con 0x7fd1a0144ad0 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd1a579a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a01580a0 con 0x7fd1a009f2b0 2023-09-29T23:06:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd1a579a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a0158360 con 0x7fd1a0063330 2023-09-29T23:06:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd19effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a006ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd19effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a006ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36680/0 (socket says 172.21.15.121:36680) 2023-09-29T23:06:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd19effd700 1 -- 172.21.15.121:0/396626643 learned_addr learned my addr 172.21.15.121:0/396626643 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fe02d545700 1 -- 172.21.15.121:0/4155497734 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024009e80 con 0x7fe0280b4ba0 2023-09-29T23:06:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fe02d545700 1 -- 172.21.15.121:0/4155497734 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02400b860 con 0x7fe0280b4ba0 2023-09-29T23:06:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fe02d545700 1 -- 172.21.15.121:0/4155497734 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02400bbe0 con 0x7fe0280b4ba0 2023-09-29T23:06:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.641+0000 7fd19e7fc700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1a009f2b0 0x7fd1a006e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd19effd700 1 -- 172.21.15.121:0/396626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1a009f2b0 msgr2=0x7fd1a006e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd19effd700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1a009f2b0 0x7fd1a006e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd19effd700 1 -- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1a0144ad0 msgr2=0x7fd1a0142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd19effd700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1a0144ad0 0x7fd1a0142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd19effd700 1 -- 172.21.15.121:0/396626643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd194009480 con 0x7fd1a0063330 2023-09-29T23:06:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd19effd700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a006ca50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0145750 tx=0x7fd194016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fb8f77fe700 1 -- 172.21.15.121:0/484912769 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8f808de30 con 0x7fb90009b2f0 2023-09-29T23:06:28.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fe0307ab700 1 -- 172.21.15.121:0/4155497734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0280b4ba0 msgr2=0x7fe0281444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fe0307ab700 1 --2- 172.21.15.121:0/4155497734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0280b4ba0 0x7fe0281444a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe0240098e0 tx=0x7fe0240091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fe0307ab700 1 -- 172.21.15.121:0/4155497734 shutdown_connections 2023-09-29T23:06:28.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fe0307ab700 1 --2- 172.21.15.121:0/4155497734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0280b4ba0 0x7fe0281444a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fe0307ab700 1 -- 172.21.15.121:0/4155497734 >> 172.21.15.121:0/4155497734 conn(0x7fe028067440 msgr2=0x7fe028067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd186ffd700 1 -- 172.21.15.121:0/396626643 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194011690 con 0x7fd1a0063330 2023-09-29T23:06:28.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd186ffd700 1 -- 172.21.15.121:0/396626643 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd194020070 con 0x7fd1a0063330 2023-09-29T23:06:28.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a006db60 con 0x7fd1a0063330 2023-09-29T23:06:28.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd186ffd700 1 -- 172.21.15.121:0/396626643 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194011a40 con 0x7fd1a0063330 2023-09-29T23:06:28.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.642+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a00a0fd0 con 0x7fd1a0063330 2023-09-29T23:06:28.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.643+0000 7fe0307ab700 1 -- 172.21.15.121:0/4155497734 shutdown_connections 2023-09-29T23:06:28.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.643+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a0071210 con 0x7fd1a0063330 2023-09-29T23:06:28.675 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.643+0000 7fe0307ab700 1 -- 172.21.15.121:0/4155497734 wait complete. 2023-09-29T23:06:28.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 Processor -- start 2023-09-29T23:06:28.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fd186ffd700 1 -- 172.21.15.121:0/396626643 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd194022030 con 0x7fd1a0063330 2023-09-29T23:06:28.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fd186ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 -- start start 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02806f960 0x7fe0280a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0280b4ba0 0x7fe0280a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe028141340 0x7fe0280a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0280a2070 con 0x7fe028141340 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0280a2c70 con 0x7fe02806f960 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fe0307ab700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0280a2260 con 0x7fe0280b4ba0 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02ed48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe028141340 0x7fe0280a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02ed48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe028141340 0x7fe0280a36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58648/0 (socket says 172.21.15.121:58648) 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02ed48700 1 -- 172.21.15.121:0/193521192 learned_addr learned my addr 172.21.15.121:0/193521192 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02dd46700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0280b4ba0 0x7fe0280a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02e547700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02806f960 0x7fe0280a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02e547700 1 -- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0280b4ba0 msgr2=0x7fe0280a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02e547700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0280b4ba0 0x7fe0280a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02e547700 1 -- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe028141340 msgr2=0x7fe0280a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02e547700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe028141340 0x7fe0280a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02e547700 1 -- 172.21.15.121:0/193521192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe024009420 con 0x7fe02806f960 2023-09-29T23:06:28.678 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fe02e547700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02806f960 0x7fe0280a0dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe0280a73c0 tx=0x7fe024017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.647+0000 7fe01e7fc700 1 -- 172.21.15.121:0/193521192 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02400b9d0 con 0x7fe02806f960 2023-09-29T23:06:28.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.647+0000 7fe01e7fc700 1 -- 172.21.15.121:0/193521192 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02400bb30 con 0x7fe02806f960 2023-09-29T23:06:28.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.647+0000 7fe01e7fc700 1 -- 172.21.15.121:0/193521192 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024025630 con 0x7fe02806f960 2023-09-29T23:06:28.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.647+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0280a2500 con 0x7fe02806f960 2023-09-29T23:06:28.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.647+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0280a3c00 con 0x7fe02806f960 2023-09-29T23:06:28.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fd186ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fd186ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fd186ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.645+0000 7fd186ffd700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd18c098850 0x7fd18c09ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fd186ffd700 1 -- 172.21.15.121:0/396626643 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fd1940c1e70 con 0x7fd1a0063330 2023-09-29T23:06:28.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.646+0000 7fd19e7fc700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd18c098850 0x7fd18c09ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.647+0000 7fd19e7fc700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd18c098850 0x7fd18c09ad00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0148d70 tx=0x7fd190008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.648+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe028063ab0 con 0x7fe02806f960 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.649+0000 7fe01e7fc700 1 -- 172.21.15.121:0/193521192 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe024023070 con 0x7fe02806f960 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.649+0000 7fe01e7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.649+0000 7fe01e7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.649+0000 7fe01e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.649+0000 7fe01e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.649+0000 7fe01e7fc700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe00c098780 0x7fe00c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:28.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.650+0000 7fe02dd46700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe00c098780 0x7fe00c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:28.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.650+0000 7fe01e7fc700 1 -- 172.21.15.121:0/193521192 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fe0240c1610 con 0x7fe02806f960 2023-09-29T23:06:28.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.650+0000 7fe02dd46700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe00c098780 0x7fe00c09ac30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe02809dec0 tx=0x7fe018007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:28.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.650+0000 7fd186ffd700 1 -- 172.21.15.121:0/396626643 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd19408f5b0 con 0x7fd1a0063330 2023-09-29T23:06:28.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.653+0000 7fe01e7fc700 1 -- 172.21.15.121:0/193521192 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe02408ecb0 con 0x7fe02806f960 2023-09-29T23:06:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.832+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdf54071660 con 0x7fdf540632a0 2023-09-29T23:06:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.832+0000 7fdf3affd700 1 -- 172.21.15.121:0/4112265372 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf44017090 con 0x7fdf540632a0 2023-09-29T23:06:28.832 INFO:teuthology.orchestra.run.smithi121.stdout:38654705672 2023-09-29T23:06:28.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.834+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb900071660 con 0x7fb90009b2f0 2023-09-29T23:06:28.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.834+0000 7fb8f77fe700 1 -- 172.21.15.121:0/484912769 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8f8091450 con 0x7fb90009b2f0 2023-09-29T23:06:28.834 INFO:teuthology.orchestra.run.smithi121.stdout:38654705672 2023-09-29T23:06:28.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.834+0000 7fdf59913700 10 mgrc shutdown 2023-09-29T23:06:28.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdf40098780 msgr2=0x7fdf4009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdf40098780 0x7fdf4009ac30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf5409c2b0 tx=0x7fdf48009040 comp rx=0 tx=0).stop 2023-09-29T23:06:28.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540632a0 msgr2=0x7fdf540a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540632a0 0x7fdf540a2460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400c490 tx=0x7fdf44007ec0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 shutdown_connections 2023-09-29T23:06:28.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdf40098780 0x7fdf4009ac30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf540a4590 0x7fdf54145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf540632a0 0x7fdf540a2460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 --2- 172.21.15.121:0/4112265372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf5409c740 0x7fdf54144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 >> 172.21.15.121:0/4112265372 conn(0x7fdf54069ec0 msgr2=0x7fdf54145f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 shutdown_connections 2023-09-29T23:06:28.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.835+0000 7fdf59913700 1 -- 172.21.15.121:0/4112265372 wait complete. 2023-09-29T23:06:28.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 10 mgrc shutdown 2023-09-29T23:06:28.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8e0098780 msgr2=0x7fb8e009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8e0098780 0x7fb8e009ac30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000c00 tx=0x7fb8e8015000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90009b2f0 msgr2=0x7fb900140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90009b2f0 0x7fb900140090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb90006bc40 tx=0x7fb8f8001cd0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 shutdown_connections 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8e0098780 0x7fb8e009ac30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90009b2f0 0x7fb900140090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8ec004d00 0x7fb900145c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 --2- 172.21.15.121:0/484912769 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000a2b60 0x7fb900099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 >> 172.21.15.121:0/484912769 conn(0x7fb9000a5bc0 msgr2=0x7fb9000b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 shutdown_connections 2023-09-29T23:06:28.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.837+0000 7fb9060ec700 1 -- 172.21.15.121:0/484912769 wait complete. 2023-09-29T23:06:28.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705672 for osd.3 2023-09-29T23:06:28.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705672 for osd.1 2023-09-29T23:06:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.877+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd1a006abd0 con 0x7fd1a0063330 2023-09-29T23:06:28.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.877+0000 7fd186ffd700 1 -- 172.21.15.121:0/396626643 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd194092bd0 con 0x7fd1a0063330 2023-09-29T23:06:28.877 INFO:teuthology.orchestra.run.smithi121.stdout:38654705670 2023-09-29T23:06:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.879+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe028063ab0 con 0x7fe02806f960 2023-09-29T23:06:28.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 10 mgrc shutdown 2023-09-29T23:06:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fe01e7fc700 1 -- 172.21.15.121:0/193521192 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe028063ab0 con 0x7fe02806f960 2023-09-29T23:06:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd18c098850 msgr2=0x7fd18c09ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd18c098850 0x7fd18c09ad00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0148d70 tx=0x7fd190008040 comp rx=0 tx=0).stop 2023-09-29T23:06:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 msgr2=0x7fd1a006ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a006ca50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0145750 tx=0x7fd194016000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.881 INFO:teuthology.orchestra.run.smithi121.stdout:38654705672 2023-09-29T23:06:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 shutdown_connections 2023-09-29T23:06:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd18c098850 0x7fd18c09ad00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1a0144ad0 0x7fd1a0142450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1a0063330 0x7fd1a006ca50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 --2- 172.21.15.121:0/396626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1a009f2b0 0x7fd1a006e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 >> 172.21.15.121:0/396626643 conn(0x7fd1a00a5bc0 msgr2=0x7fd1a009e280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 shutdown_connections 2023-09-29T23:06:28.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.880+0000 7fd1a579a700 1 -- 172.21.15.121:0/396626643 wait complete. 2023-09-29T23:06:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.882+0000 7fe0307ab700 10 mgrc shutdown 2023-09-29T23:06:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.882+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe00c098780 msgr2=0x7fe00c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.882+0000 7fe0307ab700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe00c098780 0x7fe00c09ac30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe02809dec0 tx=0x7fe018007ec0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.882+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02806f960 msgr2=0x7fe0280a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:28.883 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.882+0000 7fe0307ab700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02806f960 0x7fe0280a0dc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe0280a73c0 tx=0x7fe024017000 comp rx=0 tx=0).stop 2023-09-29T23:06:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 shutdown_connections 2023-09-29T23:06:28.885 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe00c098780 0x7fe00c09ac30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe028141340 0x7fe0280a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe02806f960 0x7fe0280a0dc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 --2- 172.21.15.121:0/193521192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0280b4ba0 0x7fe0280a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 >> 172.21.15.121:0/193521192 conn(0x7fe028067440 msgr2=0x7fe028070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 shutdown_connections 2023-09-29T23:06:28.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:28.886+0000 7fe0307ab700 1 -- 172.21.15.121:0/193521192 wait complete. 2023-09-29T23:06:28.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705670 for osd.2 2023-09-29T23:06:28.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705672 for osd.0 2023-09-29T23:06:29.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-29T23:06:29.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-29T23:06:29.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-29T23:06:29.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-29T23:06:29.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.986+0000 7ffa43a6d700 1 Processor -- start 2023-09-29T23:06:29.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.986+0000 7ffa43a6d700 1 -- start start 2023-09-29T23:06:29.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa43a6d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c0b4ba0 0x7ffa3c1444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:29.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa43a6d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffa3c141340 0x7ffa3c14af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:29.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa43a6d700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3c143d00 con 0x7ffa3c06f960 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa43a6d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3c143fc0 con 0x7ffa3c141340 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa43a6d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c0b4f70 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa41809700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c0b4ba0 0x7ffa3c1444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa41809700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c0b4ba0 0x7ffa3c1444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36696/0 (socket says 172.21.15.121:36696) 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa41809700 1 -- 172.21.15.121:0/50992457 learned_addr learned my addr 172.21.15.121:0/50992457 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.987+0000 7ffa4200a700 1 --2- 172.21.15.121:0/50992457 >> v2:172.21.15.123:3300/0 conn(0x7ffa3c141340 0x7ffa3c14af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa41809700 1 -- 172.21.15.121:0/50992457 >> v2:172.21.15.123:3300/0 conn(0x7ffa3c141340 msgr2=0x7ffa3c14af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa41809700 1 --2- 172.21.15.121:0/50992457 >> v2:172.21.15.123:3300/0 conn(0x7ffa3c141340 0x7ffa3c14af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa41809700 1 -- 172.21.15.121:0/50992457 >> v1:172.21.15.123:6789/0 conn(0x7ffa3c06f960 legacy=0x7ffa3c140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa41809700 1 -- 172.21.15.121:0/50992457 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3c1449e0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa41809700 1 --2- 172.21.15.121:0/50992457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c0b4ba0 0x7ffa3c1444a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa380099b0 tx=0x7ffa38009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf8b10c2a7c9c519 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:29.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa40807700 1 -- 172.21.15.121:0/50992457 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38009ea0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa40807700 1 -- 172.21.15.121:0/50992457 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3800b860 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.988+0000 7ffa40807700 1 -- 172.21.15.121:0/50992457 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800bc00 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 -- 172.21.15.121:0/50992457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c0b4ba0 msgr2=0x7ffa3c1444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:29.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/50992457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c0b4ba0 0x7ffa3c1444a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa380099b0 tx=0x7ffa38009250 comp rx=0 tx=0).stop 2023-09-29T23:06:29.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 -- 172.21.15.121:0/50992457 shutdown_connections 2023-09-29T23:06:29.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/50992457 >> v2:172.21.15.123:3300/0 conn(0x7ffa3c141340 0x7ffa3c14af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/50992457 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c0b4ba0 0x7ffa3c1444a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 -- 172.21.15.121:0/50992457 >> 172.21.15.121:0/50992457 conn(0x7ffa3c067440 msgr2=0x7ffa3c067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 -- 172.21.15.121:0/50992457 shutdown_connections 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.989+0000 7ffa43a6d700 1 -- 172.21.15.121:0/50992457 wait complete. 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.990+0000 7ffa43a6d700 1 Processor -- start 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.990+0000 7ffa43a6d700 1 -- start start 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa43a6d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa3c06f960 0x7ffa3c0a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa43a6d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa3c0b4ba0 0x7ffa3c0a3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa43a6d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c141340 0x7ffa3c0a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa43a6d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c0a0cc0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa43a6d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c1509a0 con 0x7ffa3c06f960 2023-09-29T23:06:29.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa43a6d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c0a3030 con 0x7ffa3c141340 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa4200a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c141340 0x7ffa3c0a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa4200a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c141340 0x7ffa3c0a3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36702/0 (socket says 172.21.15.121:36702) 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa4200a700 1 -- 172.21.15.121:0/740770525 learned_addr learned my addr 172.21.15.121:0/740770525 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa41809700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa3c06f960 0x7ffa3c0a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa41008700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa3c0b4ba0 0x7ffa3c0a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa41008700 1 -- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c141340 msgr2=0x7ffa3c0a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa41008700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c141340 0x7ffa3c0a3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa41008700 1 -- 172.21.15.121:0/740770525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa3c06f960 msgr2=0x7ffa3c0a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa41008700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa3c06f960 0x7ffa3c0a23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:29.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.991+0000 7ffa41008700 1 -- 172.21.15.121:0/740770525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa380094f0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa4200a700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c141340 0x7ffa3c0a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa41809700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa3c06f960 0x7ffa3c0a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa41008700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa3c0b4ba0 0x7ffa3c0a3810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c590 tx=0x7ffa2c007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa31ffb700 1 -- 172.21.15.121:0/740770525 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c011590 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa31ffb700 1 -- 172.21.15.121:0/740770525 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c011710 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3c150650 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa31ffb700 1 -- 172.21.15.121:0/740770525 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c011ad0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.992+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3c0a2930 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.993+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3c071660 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa31ffb700 1 -- 172.21.15.121:0/740770525 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ffa2c01b3b0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa31ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:29.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa31ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:29.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa31ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:29.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa31ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:29.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa31ffb700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ffa20098700 0x7ffa2009abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:29.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa41809700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ffa20098700 0x7ffa2009abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:29.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.995+0000 7ffa31ffb700 1 -- 172.21.15.121:0/740770525 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7ffa2c0bf7b0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:29.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:29.996+0000 7ffa41809700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ffa20098700 0x7ffa2009abb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffa380099b0 tx=0x7ffa38017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.003 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.004+0000 7ffa31ffb700 1 -- 172.21.15.121:0/740770525 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa2c08cef0 con 0x7ffa3c0b4ba0 2023-09-29T23:06:30.025 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.026+0000 7fc59c31c700 1 Processor -- start 2023-09-29T23:06:30.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.026+0000 7fc59c31c700 1 -- start start 2023-09-29T23:06:30.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.026+0000 7fc59c31c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5940b4ba0 0x7fc5941444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc59c31c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc59406f960 0x7fc594140c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc59c31c700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc594143d00 con 0x7fc5940a5440 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc59c31c700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc594143fc0 con 0x7fc59406f960 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc59c31c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5940b4f70 con 0x7fc5940b4ba0 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc59406f960 0x7fc594140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc59406f960 0x7fc594140c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58658/0 (socket says 172.21.15.121:58658) 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 -- 172.21.15.121:0/904441104 learned_addr learned my addr 172.21.15.121:0/904441104 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc59a0b8700 1 --2- 172.21.15.121:0/904441104 >> v2:172.21.15.123:3300/0 conn(0x7fc5940b4ba0 0x7fc5941444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 -- 172.21.15.121:0/904441104 >> v2:172.21.15.123:3300/0 conn(0x7fc5940b4ba0 msgr2=0x7fc5941444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 --2- 172.21.15.121:0/904441104 >> v2:172.21.15.123:3300/0 conn(0x7fc5940b4ba0 0x7fc5941444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 -- 172.21.15.121:0/904441104 >> v1:172.21.15.121:6789/0 conn(0x7fc5940a5440 legacy=0x7fc59406dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 -- 172.21.15.121:0/904441104 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5941449e0 con 0x7fc59406f960 2023-09-29T23:06:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5998b7700 1 --2- 172.21.15.121:0/904441104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc594140c30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc594067fc0 tx=0x7fc584009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c835bcd45f2cb0c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5990b6700 1 -- 172.21.15.121:0/904441104 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584009e70 con 0x7fc59406f960 2023-09-29T23:06:30.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5990b6700 1 -- 172.21.15.121:0/904441104 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58400b860 con 0x7fc59406f960 2023-09-29T23:06:30.029 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.027+0000 7fc5990b6700 1 -- 172.21.15.121:0/904441104 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58400bc00 con 0x7fc59406f960 2023-09-29T23:06:30.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 -- 172.21.15.121:0/904441104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 msgr2=0x7fc594140c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 --2- 172.21.15.121:0/904441104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc594140c30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc594067fc0 tx=0x7fc584009250 comp rx=0 tx=0).stop 2023-09-29T23:06:30.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 -- 172.21.15.121:0/904441104 shutdown_connections 2023-09-29T23:06:30.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 --2- 172.21.15.121:0/904441104 >> v2:172.21.15.123:3300/0 conn(0x7fc5940b4ba0 0x7fc5941444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.031 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 --2- 172.21.15.121:0/904441104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc594140c30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 -- 172.21.15.121:0/904441104 >> 172.21.15.121:0/904441104 conn(0x7fc594067440 msgr2=0x7fc594067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 -- 172.21.15.121:0/904441104 shutdown_connections 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.028+0000 7fc59c31c700 1 -- 172.21.15.121:0/904441104 wait complete. 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.029+0000 7fc59c31c700 1 Processor -- start 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.029+0000 7fc59c31c700 1 -- start start 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59c31c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc5940a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59c31c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5940a5440 0x7fc59413dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59c31c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5940b4ba0 0x7fc59413b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59c31c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940a0c50 con 0x7fc59406f960 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59c31c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940a37b0 con 0x7fc5940b4ba0 2023-09-29T23:06:30.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59c31c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940a3a70 con 0x7fc5940a5440 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc5940a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc5940a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58666/0 (socket says 172.21.15.121:58666) 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 -- 172.21.15.121:0/511741785 learned_addr learned my addr 172.21.15.121:0/511741785 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a8b9700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5940b4ba0 0x7fc59413b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 -- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5940a5440 msgr2=0x7fc59413dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5940a5440 0x7fc59413dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 -- 172.21.15.121:0/511741785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5940b4ba0 msgr2=0x7fc59413b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5940b4ba0 0x7fc59413b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 -- 172.21.15.121:0/511741785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5840094f0 con 0x7fc59406f960 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a8b9700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5940b4ba0 0x7fc59413b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:30.033 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.030+0000 7fc59a0b8700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc5940a2380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc59000ec80 tx=0x7fc5900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.031+0000 7fc58a7fc700 1 -- 172.21.15.121:0/511741785 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59000c850 con 0x7fc59406f960 2023-09-29T23:06:30.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.031+0000 7fc58a7fc700 1 -- 172.21.15.121:0/511741785 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59000c9b0 con 0x7fc59406f960 2023-09-29T23:06:30.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.031+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5940a3450 con 0x7fc59406f960 2023-09-29T23:06:30.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.031+0000 7fc58a7fc700 1 -- 172.21.15.121:0/511741785 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59000cd50 con 0x7fc59406f960 2023-09-29T23:06:30.034 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.031+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59409fa40 con 0x7fc59406f960 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.032+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc594071660 con 0x7fc59406f960 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7fc58a7fc700 1 -- 172.21.15.121:0/511741785 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc590013680 con 0x7fc59406f960 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7fc58a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7fc58a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7fc58a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7fc58a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7fc58a7fc700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc578098700 0x7fc57809abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7f2e49c30700 1 Processor -- start 2023-09-29T23:06:30.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7fc58a7fc700 1 -- 172.21.15.121:0/511741785 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fc5900c0810 con 0x7fc59406f960 2023-09-29T23:06:30.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.034+0000 7fc5998b7700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc578098700 0x7fc57809abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.033+0000 7f2e49c30700 1 -- start start 2023-09-29T23:06:30.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.034+0000 7f2e49c30700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2e4414e460 0x7f2e44150840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.034+0000 7f2e49c30700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e44063700 con 0x7f2e440a5080 2023-09-29T23:06:30.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.034+0000 7f2e49c30700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e440a5450 con 0x7f2e44063330 2023-09-29T23:06:30.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.034+0000 7f2e49c30700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e44145610 con 0x7f2e4414e460 2023-09-29T23:06:30.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7f2e43fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2e4414e460 0x7f2e44150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7f2e43fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2e4414e460 0x7f2e44150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58668/0 (socket says 172.21.15.121:58668) 2023-09-29T23:06:30.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7f2e43fff700 1 -- 172.21.15.121:0/1181876661 learned_addr learned my addr 172.21.15.121:0/1181876661 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:30.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7fc5998b7700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc578098700 0x7fc57809abb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc594067fc0 tx=0x7fc584017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7f2e43fff700 1 -- 172.21.15.121:0/1181876661 >> v1:172.21.15.123:6789/0 conn(0x7f2e44063330 legacy=0x7f2e4409f2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-29T23:06:30.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7f2e43fff700 1 -- 172.21.15.121:0/1181876661 >> v1:172.21.15.121:6789/0 conn(0x7f2e440a5080 legacy=0x7f2e44144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:30.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7f2e43fff700 1 -- 172.21.15.121:0/1181876661 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e441458d0 con 0x7f2e4414e460 2023-09-29T23:06:30.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.035+0000 7f2e43fff700 1 --2- 172.21.15.121:0/1181876661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 0x7f2e44150840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c01af20 tx=0x7f2e3c01a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758588514d5cdcda server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.038+0000 7f2e427fc700 1 -- 172.21.15.121:0/1181876661 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c01f070 con 0x7f2e4414e460 2023-09-29T23:06:30.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.038+0000 7f2e427fc700 1 -- 172.21.15.121:0/1181876661 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c01ca00 con 0x7f2e4414e460 2023-09-29T23:06:30.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.038+0000 7f2e427fc700 1 -- 172.21.15.121:0/1181876661 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c01a2e0 con 0x7f2e4414e460 2023-09-29T23:06:30.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.038+0000 7fc58a7fc700 1 -- 172.21.15.121:0/511741785 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc59008dec0 con 0x7fc59406f960 2023-09-29T23:06:30.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.039+0000 7f2e49c30700 1 -- 172.21.15.121:0/1181876661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 msgr2=0x7f2e44150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.039+0000 7f2e49c30700 1 --2- 172.21.15.121:0/1181876661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 0x7f2e44150840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c01af20 tx=0x7f2e3c01a840 comp rx=0 tx=0).stop 2023-09-29T23:06:30.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.039+0000 7f2e49c30700 1 -- 172.21.15.121:0/1181876661 shutdown_connections 2023-09-29T23:06:30.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.039+0000 7f2e49c30700 1 --2- 172.21.15.121:0/1181876661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 0x7f2e44150840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.039+0000 7f2e49c30700 1 -- 172.21.15.121:0/1181876661 >> 172.21.15.121:0/1181876661 conn(0x7f2e440a5bc0 msgr2=0x7f2e440b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.041+0000 7f2e49c30700 1 -- 172.21.15.121:0/1181876661 shutdown_connections 2023-09-29T23:06:30.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.041+0000 7f2e49c30700 1 -- 172.21.15.121:0/1181876661 wait complete. 2023-09-29T23:06:30.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.042+0000 7f2e49c30700 1 Processor -- start 2023-09-29T23:06:30.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.042+0000 7f2e49c30700 1 -- start start 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e49c30700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e44063330 0x7f2e4406d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e49c30700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e440a5080 0x7f2e4406cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e49c30700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 0x7f2e44147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e49c30700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e441447c0 con 0x7f2e4414e460 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e49c30700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4406d400 con 0x7f2e44063330 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e49c30700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4406deb0 con 0x7f2e440a5080 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e440a5080 0x7f2e4406cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e440a5080 0x7f2e4406cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36714/0 (socket says 172.21.15.121:36714) 2023-09-29T23:06:30.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 -- 172.21.15.121:0/676922210 learned_addr learned my addr 172.21.15.121:0/676922210 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:30.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e43fff700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 0x7f2e44147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e437fe700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e44063330 0x7f2e4406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 -- 172.21.15.121:0/676922210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e44063330 msgr2=0x7f2e4406d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e44063330 0x7f2e4406d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 -- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 msgr2=0x7f2e44147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 0x7f2e44147ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.043+0000 7f2e42ffd700 1 -- 172.21.15.121:0/676922210 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c01aae0 con 0x7f2e440a5080 2023-09-29T23:06:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.044+0000 7f2e42ffd700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e440a5080 0x7f2e4406cec0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800b600 tx=0x7f2e380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.044+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/676922210 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800f510 con 0x7f2e440a5080 2023-09-29T23:06:30.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.044+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/676922210 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3800f690 con 0x7f2e440a5080 2023-09-29T23:06:30.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.044+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4406e150 con 0x7f2e440a5080 2023-09-29T23:06:30.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.044+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e44148410 con 0x7f2e440a5080 2023-09-29T23:06:30.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.044+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/676922210 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800fa50 con 0x7f2e440a5080 2023-09-29T23:06:30.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.045+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e240050b0 con 0x7f2e440a5080 2023-09-29T23:06:30.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.047+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/676922210 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2e38024430 con 0x7f2e440a5080 2023-09-29T23:06:30.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.047+0000 7f2e2b7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:30.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.048+0000 7f2e2b7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:30.050 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.048+0000 7f2e2b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.048+0000 7f2e2b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.048+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2e300987b0 0x7f2e3009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.048+0000 7f2e437fe700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2e300987b0 0x7f2e3009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.048+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/676922210 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f2e380c0810 con 0x7f2e440a5080 2023-09-29T23:06:30.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.048+0000 7f2e437fe700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2e300987b0 0x7f2e3009ac60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009b40 tx=0x7f2e340090f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.054+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/676922210 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e3808e000 con 0x7f2e440a5080 2023-09-29T23:06:30.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.081+0000 7fb966dfd700 1 Processor -- start 2023-09-29T23:06:30.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.081+0000 7fb966dfd700 1 -- start start 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb966dfd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9600a5080 0x7fb960144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb966dfd700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb960063700 con 0x7fb960145010 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb966dfd700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9600a5450 con 0x7fb960063330 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb966dfd700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb960145610 con 0x7fb9600a5080 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb96539a700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb960145010 0x7fb96014f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44520/0 (socket says 172.21.15.121:44520) 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb96539a700 1 -- 172.21.15.121:0/1947288118 learned_addr learned my addr 172.21.15.121:0/1947288118 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95ffff700 1 --2- 172.21.15.121:0/1947288118 >> v2:172.21.15.123:3300/0 conn(0x7fb9600a5080 0x7fb960144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95f7fe700 1 -- 172.21.15.121:0/1947288118 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041776161 0 0) 0x7fb960063700 con 0x7fb960145010 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95f7fe700 1 -- 172.21.15.121:0/1947288118 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb950003570 con 0x7fb960145010 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95f7fe700 1 -- 172.21.15.121:0/1947288118 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082580019 0 0) 0x7fb9600a5450 con 0x7fb960063330 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95f7fe700 1 -- 172.21.15.121:0/1947288118 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb960063700 con 0x7fb960063330 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95ffff700 1 -- 172.21.15.121:0/1947288118 >> v1:172.21.15.123:6789/0 conn(0x7fb960063330 legacy=0x7fb96009f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95ffff700 1 -- 172.21.15.121:0/1947288118 >> v1:172.21.15.121:6789/0 conn(0x7fb960145010 legacy=0x7fb96014f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.082+0000 7fb95ffff700 1 -- 172.21.15.121:0/1947288118 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9601458d0 con 0x7fb9600a5080 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.083+0000 7fb95ffff700 1 --2- 172.21.15.121:0/1947288118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9600a5080 0x7fb960144ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0098e0 tx=0x7fb94c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50f62457dbc0e315 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.083+0000 7fb95f7fe700 1 -- 172.21.15.121:0/1947288118 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c011630 con 0x7fb9600a5080 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.083+0000 7fb95f7fe700 1 -- 172.21.15.121:0/1947288118 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c011790 con 0x7fb9600a5080 2023-09-29T23:06:30.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.083+0000 7fb95f7fe700 1 -- 172.21.15.121:0/1947288118 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c011b10 con 0x7fb9600a5080 2023-09-29T23:06:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.084+0000 7fb966dfd700 1 -- 172.21.15.121:0/1947288118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9600a5080 msgr2=0x7fb960144ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.084+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1947288118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9600a5080 0x7fb960144ad0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0098e0 tx=0x7fb94c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.084+0000 7fb966dfd700 1 -- 172.21.15.121:0/1947288118 shutdown_connections 2023-09-29T23:06:30.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.084+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1947288118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9600a5080 0x7fb960144ad0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.084+0000 7fb966dfd700 1 -- 172.21.15.121:0/1947288118 >> 172.21.15.121:0/1947288118 conn(0x7fb9600a5bc0 msgr2=0x7fb9600b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.084+0000 7fb966dfd700 1 -- 172.21.15.121:0/1947288118 shutdown_connections 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.084+0000 7fb966dfd700 1 -- 172.21.15.121:0/1947288118 wait complete. 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 Processor -- start 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 -- start start 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb960063330 0x7fb96006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600a5080 0x7fb96006d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb960145010 0x7fb96006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96006db00 con 0x7fb960063330 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96006e4c0 con 0x7fb960145010 2023-09-29T23:06:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.085+0000 7fb966dfd700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96006e690 con 0x7fb9600a5080 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb964b99700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb960063330 0x7fb96006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb960145010 0x7fb96006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb964b99700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb960063330 0x7fb96006eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58688/0 (socket says 172.21.15.121:58688) 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb964b99700 1 -- 172.21.15.121:0/1113926889 learned_addr learned my addr 172.21.15.121:0/1113926889 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb960145010 0x7fb96006d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.121:52034/0 (socket says 172.21.15.121:52034) 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb95ffff700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600a5080 0x7fb96006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 -- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600a5080 msgr2=0x7fb96006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600a5080 0x7fb96006d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 -- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb960063330 msgr2=0x7fb96006eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb960063330 0x7fb96006eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.088 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 -- 172.21.15.121:0/1113926889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c009420 con 0x7fb960145010 2023-09-29T23:06:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb96539a700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb960145010 0x7fb96006d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb95800cb90 tx=0x7fb958001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb95cff9700 1 -- 172.21.15.121:0/1113926889 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95800a7a0 con 0x7fb960145010 2023-09-29T23:06:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb95cff9700 1 -- 172.21.15.121:0/1113926889 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95800a900 con 0x7fb960145010 2023-09-29T23:06:30.089 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.086+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96006f060 con 0x7fb960145010 2023-09-29T23:06:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.087+0000 7fb95cff9700 1 -- 172.21.15.121:0/1113926889 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95800acc0 con 0x7fb960145010 2023-09-29T23:06:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.087+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9600a15d0 con 0x7fb960145010 2023-09-29T23:06:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.088+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96006abd0 con 0x7fb960145010 2023-09-29T23:06:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.089+0000 7fb95cff9700 1 -- 172.21.15.121:0/1113926889 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb958007300 con 0x7fb960145010 2023-09-29T23:06:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.089+0000 7fb95cff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.090+0000 7fb95cff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:30.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.090+0000 7fb95cff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.090+0000 7fb95cff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.090+0000 7fb95cff9700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb944098800 0x7fb94409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.090+0000 7fb95cff9700 1 -- 172.21.15.121:0/1113926889 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fb9580c0630 con 0x7fb960145010 2023-09-29T23:06:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.090+0000 7fb964b99700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb944098800 0x7fb94409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.090+0000 7fb964b99700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb944098800 0x7fb94409acb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb954007fd0 tx=0x7fb954007db0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.094+0000 7fb95cff9700 1 -- 172.21.15.121:0/1113926889 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb95808dcd0 con 0x7fb960145010 2023-09-29T23:06:30.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.211+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffa3c0a2b50 con 0x7ffa3c0b4ba0 2023-09-29T23:06:30.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.212+0000 7ffa31ffb700 1 -- 172.21.15.121:0/740770525 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa2c017020 con 0x7ffa3c0b4ba0 2023-09-29T23:06:30.212 INFO:teuthology.orchestra.run.smithi121.stdout:38654705674 2023-09-29T23:06:30.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 10 mgrc shutdown 2023-09-29T23:06:30.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ffa20098700 msgr2=0x7ffa2009abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ffa20098700 0x7ffa2009abb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffa380099b0 tx=0x7ffa38017000 comp rx=0 tx=0).stop 2023-09-29T23:06:30.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa3c0b4ba0 msgr2=0x7ffa3c0a3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa3c0b4ba0 0x7ffa3c0a3810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c590 tx=0x7ffa2c007ec0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 shutdown_connections 2023-09-29T23:06:30.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ffa20098700 0x7ffa2009abb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa3c141340 0x7ffa3c0a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa3c06f960 0x7ffa3c0a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 --2- 172.21.15.121:0/740770525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa3c0b4ba0 0x7ffa3c0a3810 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 >> 172.21.15.121:0/740770525 conn(0x7ffa3c067440 msgr2=0x7ffa3c0b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 shutdown_connections 2023-09-29T23:06:30.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.214+0000 7ffa43a6d700 1 -- 172.21.15.121:0/740770525 wait complete. 2023-09-29T23:06:30.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.3 2023-09-29T23:06:30.226 DEBUG:teuthology.parallel:result is None 2023-09-29T23:06:30.234 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.235+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc5941405d0 con 0x7fc59406f960 2023-09-29T23:06:30.235 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.235+0000 7fc58a7fc700 1 -- 172.21.15.121:0/511741785 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc590021090 con 0x7fc59406f960 2023-09-29T23:06:30.235 INFO:teuthology.orchestra.run.smithi121.stdout:38654705674 2023-09-29T23:06:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 10 mgrc shutdown 2023-09-29T23:06:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc578098700 msgr2=0x7fc57809abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc578098700 0x7fc57809abb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc594067fc0 tx=0x7fc584017000 comp rx=0 tx=0).stop 2023-09-29T23:06:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 msgr2=0x7fc5940a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc5940a2380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc59000ec80 tx=0x7fc5900091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 shutdown_connections 2023-09-29T23:06:30.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc578098700 0x7fc57809abb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5940b4ba0 0x7fc59413b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc59406f960 0x7fc5940a2380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 --2- 172.21.15.121:0/511741785 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5940a5440 0x7fc59413dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 >> 172.21.15.121:0/511741785 conn(0x7fc594067440 msgr2=0x7fc59413fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 shutdown_connections 2023-09-29T23:06:30.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.237+0000 7fc59c31c700 1 -- 172.21.15.121:0/511741785 wait complete. 2023-09-29T23:06:30.248 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2023-09-29T23:06:30.249 DEBUG:teuthology.parallel:result is None 2023-09-29T23:06:30.281 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.281+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2e24005630 con 0x7f2e440a5080 2023-09-29T23:06:30.282 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.282+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/676922210 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e38091620 con 0x7f2e440a5080 2023-09-29T23:06:30.282 INFO:teuthology.orchestra.run.smithi121.stdout:38654705672 2023-09-29T23:06:30.284 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 10 mgrc shutdown 2023-09-29T23:06:30.284 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2e300987b0 msgr2=0x7f2e3009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.284 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2e300987b0 0x7f2e3009ac60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009b40 tx=0x7f2e340090f0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e440a5080 msgr2=0x7f2e4406cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e440a5080 0x7f2e4406cec0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800b600 tx=0x7f2e380091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 shutdown_connections 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2e300987b0 0x7f2e3009ac60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e4414e460 0x7f2e44147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e44063330 0x7f2e4406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 --2- 172.21.15.121:0/676922210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e440a5080 0x7f2e4406cec0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 >> 172.21.15.121:0/676922210 conn(0x7f2e440a5bc0 msgr2=0x7f2e440b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 shutdown_connections 2023-09-29T23:06:30.285 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.284+0000 7f2e49c30700 1 -- 172.21.15.121:0/676922210 wait complete. 2023-09-29T23:06:30.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.2 2023-09-29T23:06:30.296 DEBUG:teuthology.parallel:result is None 2023-09-29T23:06:30.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.309+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb960071660 con 0x7fb960145010 2023-09-29T23:06:30.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.310+0000 7fb95cff9700 1 -- 172.21.15.121:0/1113926889 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb9580912f0 con 0x7fb960145010 2023-09-29T23:06:30.310 INFO:teuthology.orchestra.run.smithi121.stdout:38654705674 2023-09-29T23:06:30.313 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 10 mgrc shutdown 2023-09-29T23:06:30.313 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb944098800 msgr2=0x7fb94409acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.313 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb944098800 0x7fb94409acb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb954007fd0 tx=0x7fb954007db0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb960145010 msgr2=0x7fb96006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb960145010 0x7fb96006d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb95800cb90 tx=0x7fb958001980 comp rx=0 tx=0).stop 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 shutdown_connections 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb944098800 0x7fb94409acb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb960145010 0x7fb96006d5c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb960063330 0x7fb96006eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 --2- 172.21.15.121:0/1113926889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600a5080 0x7fb96006d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.312+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 >> 172.21.15.121:0/1113926889 conn(0x7fb9600a5bc0 msgr2=0x7fb9600b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.313+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 shutdown_connections 2023-09-29T23:06:30.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.313+0000 7fb966dfd700 1 -- 172.21.15.121:0/1113926889 wait complete. 2023-09-29T23:06:30.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2023-09-29T23:06:30.325 DEBUG:teuthology.parallel:result is None 2023-09-29T23:06:30.325 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-29T23:06:30.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:06:30.481 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.481+0000 7fccf7d99700 1 Processor -- start 2023-09-29T23:06:30.481 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf7d99700 1 -- start start 2023-09-29T23:06:30.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf7d99700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf009ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf7d99700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf01429d0 con 0x7fccf009b340 2023-09-29T23:06:30.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf7d99700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf0142c90 con 0x7fccf00630e0 2023-09-29T23:06:30.483 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf7d99700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0142710 con 0x7fccf00634b0 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf6336700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fccf009b340 0x7fccf0143f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44524/0 (socket says 172.21.15.121:44524) 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf6336700 1 -- 172.21.15.121:0/3786473521 learned_addr learned my addr 172.21.15.121:0/3786473521 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf5334700 1 --2- 172.21.15.121:0/3786473521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf009ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf4b33700 1 -- 172.21.15.121:0/3786473521 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490164425 0 0) 0x7fccf01429d0 con 0x7fccf009b340 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.482+0000 7fccf4b33700 1 -- 172.21.15.121:0/3786473521 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc003570 con 0x7fccf009b340 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf5334700 1 -- 172.21.15.121:0/3786473521 >> v1:172.21.15.123:6789/0 conn(0x7fccf00630e0 legacy=0x7fccf0142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf5334700 1 -- 172.21.15.121:0/3786473521 >> v1:172.21.15.121:6789/0 conn(0x7fccf009b340 legacy=0x7fccf0143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf5334700 1 -- 172.21.15.121:0/3786473521 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf009b7a0 con 0x7fccf00634b0 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf5334700 1 --2- 172.21.15.121:0/3786473521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf009ae00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccec009910 tx=0x7fccec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a8ff9bf5a6b8cc1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf4b33700 1 -- 172.21.15.121:0/3786473521 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec00b680 con 0x7fccf00634b0 2023-09-29T23:06:30.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf4b33700 1 -- 172.21.15.121:0/3786473521 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec00b7e0 con 0x7fccf00634b0 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf4b33700 1 -- 172.21.15.121:0/3786473521 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec00bba0 con 0x7fccf00634b0 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf7d99700 1 -- 172.21.15.121:0/3786473521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 msgr2=0x7fccf009ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf7d99700 1 --2- 172.21.15.121:0/3786473521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf009ae00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccec009910 tx=0x7fccec0091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf7d99700 1 -- 172.21.15.121:0/3786473521 shutdown_connections 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf7d99700 1 --2- 172.21.15.121:0/3786473521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf009ae00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf7d99700 1 -- 172.21.15.121:0/3786473521 >> 172.21.15.121:0/3786473521 conn(0x7fccf00a5bc0 msgr2=0x7fccf00b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf7d99700 1 -- 172.21.15.121:0/3786473521 shutdown_connections 2023-09-29T23:06:30.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.483+0000 7fccf7d99700 1 -- 172.21.15.121:0/3786473521 wait complete. 2023-09-29T23:06:30.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.484+0000 7fccf7d99700 1 Processor -- start 2023-09-29T23:06:30.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.484+0000 7fccf7d99700 1 -- start start 2023-09-29T23:06:30.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf7d99700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccf00630e0 0x7fccf00a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf7d99700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf0134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf7d99700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf009b340 0x7fccf01350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf7d99700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0135600 con 0x7fccf009b340 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf7d99700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01357e0 con 0x7fccf00630e0 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf7d99700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00a1010 con 0x7fccf00634b0 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf5334700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf0134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf6336700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf009b340 0x7fccf01350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf5334700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf0134b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36744/0 (socket says 172.21.15.121:36744) 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf5334700 1 -- 172.21.15.121:0/1087718440 learned_addr learned my addr 172.21.15.121:0/1087718440 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf5b35700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccf00630e0 0x7fccf00a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf6336700 1 -- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 msgr2=0x7fccf0134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf6336700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf0134b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf6336700 1 -- 172.21.15.121:0/1087718440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccf00630e0 msgr2=0x7fccf00a6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf6336700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccf00630e0 0x7fccf00a6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf6336700 1 -- 172.21.15.121:0/1087718440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec009420 con 0x7fccf009b340 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf5334700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf0134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.485+0000 7fccf6336700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf009b340 0x7fccf01350c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcce800cbf0 tx=0x7fcce8001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.486+0000 7fcce5ffb700 1 -- 172.21.15.121:0/1087718440 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce800a870 con 0x7fccf009b340 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.486+0000 7fcce5ffb700 1 -- 172.21.15.121:0/1087718440 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce800a9d0 con 0x7fccf009b340 2023-09-29T23:06:30.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.486+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf009f600 con 0x7fccf009b340 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.486+0000 7fcce5ffb700 1 -- 172.21.15.121:0/1087718440 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce800ad50 con 0x7fccf009b340 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.486+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf0133820 con 0x7fccf009b340 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.487+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf006abd0 con 0x7fccf009b340 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.487+0000 7fcce5ffb700 1 -- 172.21.15.121:0/1087718440 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcce8007300 con 0x7fccf009b340 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.487+0000 7fcce5ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.488+0000 7fcce5ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.488+0000 7fcce5ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.488+0000 7fcce5ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.488+0000 7fcce5ffb700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fccd4098780 0x7fccd409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.488+0000 7fccf5b35700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fccd4098780 0x7fccd409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.488+0000 7fcce5ffb700 1 -- 172.21.15.121:0/1087718440 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fcce80c0740 con 0x7fccf009b340 2023-09-29T23:06:30.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.488+0000 7fccf5b35700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fccd4098780 0x7fccd409ac30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcce0003a20 tx=0x7fcce0007f10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.492+0000 7fcce5ffb700 1 -- 172.21.15.121:0/1087718440 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcce808ddf0 con 0x7fccf009b340 2023-09-29T23:06:30.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.669+0000 7fccf7d99700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-29T23:06:30.669 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.669+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 --> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccf00a2a20 con 0x7fccd4098780 2023-09-29T23:06:30.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.676+0000 7fcce5ffb700 1 -- 172.21.15.121:0/1087718440 <== mgr.4107 v2:172.21.15.121:6808/113227 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+269404 (secure 0 0 0) 0x7fcce00026d0 con 0x7fccd4098780 2023-09-29T23:06:30.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.676+0000 7fcce5ffb700 20 mgrc handle_command_reply tid 0 r 0 2023-09-29T23:06:30.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:30.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 10 mgrc shutdown 2023-09-29T23:06:30.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fccd4098780 msgr2=0x7fccd409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fccd4098780 0x7fccd409ac30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcce0003a20 tx=0x7fcce0007f10 comp rx=0 tx=0).stop 2023-09-29T23:06:30.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf009b340 msgr2=0x7fccf01350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf009b340 0x7fccf01350c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcce800cbf0 tx=0x7fcce8001a20 comp rx=0 tx=0).stop 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 shutdown_connections 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fccd4098780 0x7fccd409ac30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf009b340 0x7fccf01350c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccf00630e0 0x7fccf00a6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 --2- 172.21.15.121:0/1087718440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf00634b0 0x7fccf0134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 >> 172.21.15.121:0/1087718440 conn(0x7fccf00a5bc0 msgr2=0x7fccf00b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 shutdown_connections 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.679+0000 7fccf7d99700 1 -- 172.21.15.121:0/1087718440 wait complete. 2023-09-29T23:06:30.682 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2023-09-29T23:06:30.691 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-09-29T23:06:29.488102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":35136,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":377452224,"statfs":{"total":386547056640,"available":386511077376,"internally_reserved":0,"allocated":884736,"data_stored":1426840,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.228406"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278366+0000","last_change":"2023-09-29T23:06:18.276673+0000","last_active":"2023-09-29T23:06:18.278366+0000","last_peered":"2023-09-29T23:06:18.278366+0000","last_clean":"2023-09-29T23:06:18.278366+0000","last_became_active":"2023-09-29T23:06:18.276312+0000","last_became_peered":"2023-09-29T23:06:18.276312+0000","last_unstale":"2023-09-29T23:06:18.278366+0000","last_undegraded":"2023-09-29T23:06:18.278366+0000","last_fullsized":"2023-09-29T23:06:18.278366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:40.183958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283354+0000","last_change":"2023-09-29T23:06:18.282307+0000","last_active":"2023-09-29T23:06:18.283354+0000","last_peered":"2023-09-29T23:06:18.283354+0000","last_clean":"2023-09-29T23:06:18.283354+0000","last_became_active":"2023-09-29T23:06:18.282064+0000","last_became_peered":"2023-09-29T23:06:18.282064+0000","last_unstale":"2023-09-29T23:06:18.283354+0000","last_undegraded":"2023-09-29T23:06:18.283354+0000","last_fullsized":"2023-09-29T23:06:18.283354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:10:08.336407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283157+0000","last_change":"2023-09-29T23:06:18.281514+0000","last_active":"2023-09-29T23:06:18.283157+0000","last_peered":"2023-09-29T23:06:18.283157+0000","last_clean":"2023-09-29T23:06:18.283157+0000","last_became_active":"2023-09-29T23:06:18.281144+0000","last_became_peered":"2023-09-29T23:06:18.281144+0000","last_unstale":"2023-09-29T23:06:18.283157+0000","last_undegraded":"2023-09-29T23:06:18.283157+0000","last_fullsized":"2023-09-29T23:06:18.283157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290407+0000","last_change":"2023-09-29T23:06:18.287192+0000","last_active":"2023-09-29T23:06:18.290407+0000","last_peered":"2023-09-29T23:06:18.290407+0000","last_clean":"2023-09-29T23:06:18.290407+0000","last_became_active":"2023-09-29T23:06:18.286917+0000","last_became_peered":"2023-09-29T23:06:18.286917+0000","last_unstale":"2023-09-29T23:06:18.290407+0000","last_undegraded":"2023-09-29T23:06:18.290407+0000","last_fullsized":"2023-09-29T23:06:18.290407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292437+0000","last_change":"2023-09-29T23:06:18.289803+0000","last_active":"2023-09-29T23:06:18.292437+0000","last_peered":"2023-09-29T23:06:18.292437+0000","last_clean":"2023-09-29T23:06:18.292437+0000","last_became_active":"2023-09-29T23:06:18.289379+0000","last_became_peered":"2023-09-29T23:06:18.289379+0000","last_unstale":"2023-09-29T23:06:18.292437+0000","last_undegraded":"2023-09-29T23:06:18.292437+0000","last_fullsized":"2023-09-29T23:06:18.292437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287884+0000","last_change":"2023-09-29T23:06:18.285440+0000","last_active":"2023-09-29T23:06:18.287884+0000","last_peered":"2023-09-29T23:06:18.287884+0000","last_clean":"2023-09-29T23:06:18.287884+0000","last_became_active":"2023-09-29T23:06:18.285182+0000","last_became_peered":"2023-09-29T23:06:18.285182+0000","last_unstale":"2023-09-29T23:06:18.287884+0000","last_undegraded":"2023-09-29T23:06:18.287884+0000","last_fullsized":"2023-09-29T23:06:18.287884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:04:37.136451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288766+0000","last_change":"2023-09-29T23:06:18.286007+0000","last_active":"2023-09-29T23:06:18.288766+0000","last_peered":"2023-09-29T23:06:18.288766+0000","last_clean":"2023-09-29T23:06:18.288766+0000","last_became_active":"2023-09-29T23:06:18.284765+0000","last_became_peered":"2023-09-29T23:06:18.284765+0000","last_unstale":"2023-09-29T23:06:18.288766+0000","last_undegraded":"2023-09-29T23:06:18.288766+0000","last_fullsized":"2023-09-29T23:06:18.288766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273887+0000","last_change":"2023-09-29T23:06:18.272963+0000","last_active":"2023-09-29T23:06:18.273887+0000","last_peered":"2023-09-29T23:06:18.273887+0000","last_clean":"2023-09-29T23:06:18.273887+0000","last_became_active":"2023-09-29T23:06:18.272736+0000","last_became_peered":"2023-09-29T23:06:18.272736+0000","last_unstale":"2023-09-29T23:06:18.273887+0000","last_undegraded":"2023-09-29T23:06:18.273887+0000","last_fullsized":"2023-09-29T23:06:18.273887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278347+0000","last_change":"2023-09-29T23:06:18.276308+0000","last_active":"2023-09-29T23:06:18.278347+0000","last_peered":"2023-09-29T23:06:18.278347+0000","last_clean":"2023-09-29T23:06:18.278347+0000","last_became_active":"2023-09-29T23:06:18.276056+0000","last_became_peered":"2023-09-29T23:06:18.276056+0000","last_unstale":"2023-09-29T23:06:18.278347+0000","last_undegraded":"2023-09-29T23:06:18.278347+0000","last_fullsized":"2023-09-29T23:06:18.278347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:36.204055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287088+0000","last_change":"2023-09-29T23:06:18.280125+0000","last_active":"2023-09-29T23:06:18.287088+0000","last_peered":"2023-09-29T23:06:18.287088+0000","last_clean":"2023-09-29T23:06:18.287088+0000","last_became_active":"2023-09-29T23:06:18.279830+0000","last_became_peered":"2023-09-29T23:06:18.279830+0000","last_unstale":"2023-09-29T23:06:18.287088+0000","last_undegraded":"2023-09-29T23:06:18.287088+0000","last_fullsized":"2023-09-29T23:06:18.287088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280206+0000","last_change":"2023-09-29T23:06:18.270148+0000","last_active":"2023-09-29T23:06:18.280206+0000","last_peered":"2023-09-29T23:06:18.280206+0000","last_clean":"2023-09-29T23:06:18.280206+0000","last_became_active":"2023-09-29T23:06:18.269859+0000","last_became_peered":"2023-09-29T23:06:18.269859+0000","last_unstale":"2023-09-29T23:06:18.280206+0000","last_undegraded":"2023-09-29T23:06:18.280206+0000","last_fullsized":"2023-09-29T23:06:18.280206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277323+0000","last_change":"2023-09-29T23:06:18.272855+0000","last_active":"2023-09-29T23:06:18.277323+0000","last_peered":"2023-09-29T23:06:18.277323+0000","last_clean":"2023-09-29T23:06:18.277323+0000","last_became_active":"2023-09-29T23:06:18.272591+0000","last_became_peered":"2023-09-29T23:06:18.272591+0000","last_unstale":"2023-09-29T23:06:18.277323+0000","last_undegraded":"2023-09-29T23:06:18.277323+0000","last_fullsized":"2023-09-29T23:06:18.277323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:35.444407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285404+0000","last_change":"2023-09-29T23:06:18.283392+0000","last_active":"2023-09-29T23:06:18.285404+0000","last_peered":"2023-09-29T23:06:18.285404+0000","last_clean":"2023-09-29T23:06:18.285404+0000","last_became_active":"2023-09-29T23:06:18.282197+0000","last_became_peered":"2023-09-29T23:06:18.282197+0000","last_unstale":"2023-09-29T23:06:18.285404+0000","last_undegraded":"2023-09-29T23:06:18.285404+0000","last_fullsized":"2023-09-29T23:06:18.285404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:33.387327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269370+0000","last_change":"2023-09-29T23:06:18.268361+0000","last_active":"2023-09-29T23:06:18.269370+0000","last_peered":"2023-09-29T23:06:18.269370+0000","last_clean":"2023-09-29T23:06:18.269370+0000","last_became_active":"2023-09-29T23:06:18.268085+0000","last_became_peered":"2023-09-29T23:06:18.268085+0000","last_unstale":"2023-09-29T23:06:18.269370+0000","last_undegraded":"2023-09-29T23:06:18.269370+0000","last_fullsized":"2023-09-29T23:06:18.269370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283344+0000","last_change":"2023-09-29T23:06:18.282177+0000","last_active":"2023-09-29T23:06:18.283344+0000","last_peered":"2023-09-29T23:06:18.283344+0000","last_clean":"2023-09-29T23:06:18.283344+0000","last_became_active":"2023-09-29T23:06:18.281841+0000","last_became_peered":"2023-09-29T23:06:18.281841+0000","last_unstale":"2023-09-29T23:06:18.283344+0000","last_undegraded":"2023-09-29T23:06:18.283344+0000","last_fullsized":"2023-09-29T23:06:18.283344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273852+0000","last_change":"2023-09-29T23:06:18.272693+0000","last_active":"2023-09-29T23:06:18.273852+0000","last_peered":"2023-09-29T23:06:18.273852+0000","last_clean":"2023-09-29T23:06:18.273852+0000","last_became_active":"2023-09-29T23:06:18.272433+0000","last_became_peered":"2023-09-29T23:06:18.272433+0000","last_unstale":"2023-09-29T23:06:18.273852+0000","last_undegraded":"2023-09-29T23:06:18.273852+0000","last_fullsized":"2023-09-29T23:06:18.273852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280760+0000","last_change":"2023-09-29T23:06:18.275372+0000","last_active":"2023-09-29T23:06:18.280760+0000","last_peered":"2023-09-29T23:06:18.280760+0000","last_clean":"2023-09-29T23:06:18.280760+0000","last_became_active":"2023-09-29T23:06:18.274861+0000","last_became_peered":"2023-09-29T23:06:18.274861+0000","last_unstale":"2023-09-29T23:06:18.280760+0000","last_undegraded":"2023-09-29T23:06:18.280760+0000","last_fullsized":"2023-09-29T23:06:18.280760+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290310+0000","last_change":"2023-09-29T23:06:18.287098+0000","last_active":"2023-09-29T23:06:18.290310+0000","last_peered":"2023-09-29T23:06:18.290310+0000","last_clean":"2023-09-29T23:06:18.290310+0000","last_became_active":"2023-09-29T23:06:18.286847+0000","last_became_peered":"2023-09-29T23:06:18.286847+0000","last_unstale":"2023-09-29T23:06:18.290310+0000","last_undegraded":"2023-09-29T23:06:18.290310+0000","last_fullsized":"2023-09-29T23:06:18.290310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277206+0000","last_change":"2023-09-29T23:06:18.273575+0000","last_active":"2023-09-29T23:06:18.277206+0000","last_peered":"2023-09-29T23:06:18.277206+0000","last_clean":"2023-09-29T23:06:18.277206+0000","last_became_active":"2023-09-29T23:06:18.273327+0000","last_became_peered":"2023-09-29T23:06:18.273327+0000","last_unstale":"2023-09-29T23:06:18.277206+0000","last_undegraded":"2023-09-29T23:06:18.277206+0000","last_fullsized":"2023-09-29T23:06:18.277206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:29:05.706848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280438+0000","last_change":"2023-09-29T23:06:18.271980+0000","last_active":"2023-09-29T23:06:18.280438+0000","last_peered":"2023-09-29T23:06:18.280438+0000","last_clean":"2023-09-29T23:06:18.280438+0000","last_became_active":"2023-09-29T23:06:18.271725+0000","last_became_peered":"2023-09-29T23:06:18.271725+0000","last_unstale":"2023-09-29T23:06:18.280438+0000","last_undegraded":"2023-09-29T23:06:18.280438+0000","last_fullsized":"2023-09-29T23:06:18.280438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283120+0000","last_change":"2023-09-29T23:06:18.281173+0000","last_active":"2023-09-29T23:06:18.283120+0000","last_peered":"2023-09-29T23:06:18.283120+0000","last_clean":"2023-09-29T23:06:18.283120+0000","last_became_active":"2023-09-29T23:06:18.280834+0000","last_became_peered":"2023-09-29T23:06:18.280834+0000","last_unstale":"2023-09-29T23:06:18.283120+0000","last_undegraded":"2023-09-29T23:06:18.283120+0000","last_fullsized":"2023-09-29T23:06:18.283120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:02:54.551488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280253+0000","last_change":"2023-09-29T23:06:18.270925+0000","last_active":"2023-09-29T23:06:18.280253+0000","last_peered":"2023-09-29T23:06:18.280253+0000","last_clean":"2023-09-29T23:06:18.280253+0000","last_became_active":"2023-09-29T23:06:18.269605+0000","last_became_peered":"2023-09-29T23:06:18.269605+0000","last_unstale":"2023-09-29T23:06:18.280253+0000","last_undegraded":"2023-09-29T23:06:18.280253+0000","last_fullsized":"2023-09-29T23:06:18.280253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287559+0000","last_change":"2023-09-29T23:06:18.284188+0000","last_active":"2023-09-29T23:06:18.287559+0000","last_peered":"2023-09-29T23:06:18.287559+0000","last_clean":"2023-09-29T23:06:18.287559+0000","last_became_active":"2023-09-29T23:06:18.283875+0000","last_became_peered":"2023-09-29T23:06:18.283875+0000","last_unstale":"2023-09-29T23:06:18.287559+0000","last_undegraded":"2023-09-29T23:06:18.287559+0000","last_fullsized":"2023-09-29T23:06:18.287559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283044+0000","last_change":"2023-09-29T23:06:18.280077+0000","last_active":"2023-09-29T23:06:18.283044+0000","last_peered":"2023-09-29T23:06:18.283044+0000","last_clean":"2023-09-29T23:06:18.283044+0000","last_became_active":"2023-09-29T23:06:18.279865+0000","last_became_peered":"2023-09-29T23:06:18.279865+0000","last_unstale":"2023-09-29T23:06:18.283044+0000","last_undegraded":"2023-09-29T23:06:18.283044+0000","last_fullsized":"2023-09-29T23:06:18.283044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:48.644161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280240+0000","last_change":"2023-09-29T23:06:18.270238+0000","last_active":"2023-09-29T23:06:18.280240+0000","last_peered":"2023-09-29T23:06:18.280240+0000","last_clean":"2023-09-29T23:06:18.280240+0000","last_became_active":"2023-09-29T23:06:18.269996+0000","last_became_peered":"2023-09-29T23:06:18.269996+0000","last_unstale":"2023-09-29T23:06:18.280240+0000","last_undegraded":"2023-09-29T23:06:18.280240+0000","last_fullsized":"2023-09-29T23:06:18.280240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287050+0000","last_change":"2023-09-29T23:06:18.279595+0000","last_active":"2023-09-29T23:06:18.287050+0000","last_peered":"2023-09-29T23:06:18.287050+0000","last_clean":"2023-09-29T23:06:18.287050+0000","last_became_active":"2023-09-29T23:06:18.278827+0000","last_became_peered":"2023-09-29T23:06:18.278827+0000","last_unstale":"2023-09-29T23:06:18.287050+0000","last_undegraded":"2023-09-29T23:06:18.287050+0000","last_fullsized":"2023-09-29T23:06:18.287050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288146+0000","last_change":"2023-09-29T23:06:18.287081+0000","last_active":"2023-09-29T23:06:18.288146+0000","last_peered":"2023-09-29T23:06:18.288146+0000","last_clean":"2023-09-29T23:06:18.288146+0000","last_became_active":"2023-09-29T23:06:18.286853+0000","last_became_peered":"2023-09-29T23:06:18.286853+0000","last_unstale":"2023-09-29T23:06:18.288146+0000","last_undegraded":"2023-09-29T23:06:18.288146+0000","last_fullsized":"2023-09-29T23:06:18.288146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:00.791418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280662+0000","last_change":"2023-09-29T23:06:17.886342+0000","last_active":"2023-09-29T23:06:18.280662+0000","last_peered":"2023-09-29T23:06:18.280662+0000","last_clean":"2023-09-29T23:06:18.280662+0000","last_became_active":"2023-09-29T23:06:17.886089+0000","last_became_peered":"2023-09-29T23:06:17.886089+0000","last_unstale":"2023-09-29T23:06:18.280662+0000","last_undegraded":"2023-09-29T23:06:18.280662+0000","last_fullsized":"2023-09-29T23:06:18.280662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:04.152750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288614+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288301+0000","last_became_peered":"2023-09-29T23:06:18.288301+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261493+0000","last_change":"2023-09-29T23:06:17.891522+0000","last_active":"2023-09-29T23:06:18.261493+0000","last_peered":"2023-09-29T23:06:18.261493+0000","last_clean":"2023-09-29T23:06:18.261493+0000","last_became_active":"2023-09-29T23:06:17.891232+0000","last_became_peered":"2023-09-29T23:06:17.891232+0000","last_unstale":"2023-09-29T23:06:18.261493+0000","last_undegraded":"2023-09-29T23:06:18.261493+0000","last_fullsized":"2023-09-29T23:06:18.261493+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287070+0000","last_change":"2023-09-29T23:06:18.279774+0000","last_active":"2023-09-29T23:06:18.287070+0000","last_peered":"2023-09-29T23:06:18.287070+0000","last_clean":"2023-09-29T23:06:18.287070+0000","last_became_active":"2023-09-29T23:06:18.279014+0000","last_became_peered":"2023-09-29T23:06:18.279014+0000","last_unstale":"2023-09-29T23:06:18.287070+0000","last_undegraded":"2023-09-29T23:06:18.287070+0000","last_fullsized":"2023-09-29T23:06:18.287070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273886+0000","last_change":"2023-09-29T23:06:17.894216+0000","last_active":"2023-09-29T23:06:18.273886+0000","last_peered":"2023-09-29T23:06:18.273886+0000","last_clean":"2023-09-29T23:06:18.273886+0000","last_became_active":"2023-09-29T23:06:17.893923+0000","last_became_peered":"2023-09-29T23:06:17.893923+0000","last_unstale":"2023-09-29T23:06:18.273886+0000","last_undegraded":"2023-09-29T23:06:18.273886+0000","last_fullsized":"2023-09-29T23:06:18.273886+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282454+0000","last_change":"2023-09-29T23:06:18.276320+0000","last_active":"2023-09-29T23:06:18.282454+0000","last_peered":"2023-09-29T23:06:18.282454+0000","last_clean":"2023-09-29T23:06:18.282454+0000","last_became_active":"2023-09-29T23:06:18.276049+0000","last_became_peered":"2023-09-29T23:06:18.276049+0000","last_unstale":"2023-09-29T23:06:18.282454+0000","last_undegraded":"2023-09-29T23:06:18.282454+0000","last_fullsized":"2023-09-29T23:06:18.282454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:32.688280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261797+0000","last_change":"2023-09-29T23:06:17.883845+0000","last_active":"2023-09-29T23:06:18.261797+0000","last_peered":"2023-09-29T23:06:18.261797+0000","last_clean":"2023-09-29T23:06:18.261797+0000","last_became_active":"2023-09-29T23:06:17.883559+0000","last_became_peered":"2023-09-29T23:06:17.883559+0000","last_unstale":"2023-09-29T23:06:18.261797+0000","last_undegraded":"2023-09-29T23:06:18.261797+0000","last_fullsized":"2023-09-29T23:06:18.261797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286481+0000","last_change":"2023-09-29T23:06:18.284038+0000","last_active":"2023-09-29T23:06:18.286481+0000","last_peered":"2023-09-29T23:06:18.286481+0000","last_clean":"2023-09-29T23:06:18.286481+0000","last_became_active":"2023-09-29T23:06:18.283764+0000","last_became_peered":"2023-09-29T23:06:18.283764+0000","last_unstale":"2023-09-29T23:06:18.286481+0000","last_undegraded":"2023-09-29T23:06:18.286481+0000","last_fullsized":"2023-09-29T23:06:18.286481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:05:42.345774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261094+0000","last_change":"2023-09-29T23:06:17.886414+0000","last_active":"2023-09-29T23:06:18.261094+0000","last_peered":"2023-09-29T23:06:18.261094+0000","last_clean":"2023-09-29T23:06:18.261094+0000","last_became_active":"2023-09-29T23:06:17.886124+0000","last_became_peered":"2023-09-29T23:06:17.886124+0000","last_unstale":"2023-09-29T23:06:18.261094+0000","last_undegraded":"2023-09-29T23:06:18.261094+0000","last_fullsized":"2023-09-29T23:06:18.261094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:32.385528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288784+0000","last_change":"2023-09-29T23:06:18.286164+0000","last_active":"2023-09-29T23:06:18.288784+0000","last_peered":"2023-09-29T23:06:18.288784+0000","last_clean":"2023-09-29T23:06:18.288784+0000","last_became_active":"2023-09-29T23:06:18.285518+0000","last_became_peered":"2023-09-29T23:06:18.285518+0000","last_unstale":"2023-09-29T23:06:18.288784+0000","last_undegraded":"2023-09-29T23:06:18.288784+0000","last_fullsized":"2023-09-29T23:06:18.288784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:54:18.859428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286862+0000","last_change":"2023-09-29T23:06:17.889196+0000","last_active":"2023-09-29T23:06:18.286862+0000","last_peered":"2023-09-29T23:06:18.286862+0000","last_clean":"2023-09-29T23:06:18.286862+0000","last_became_active":"2023-09-29T23:06:17.888932+0000","last_became_peered":"2023-09-29T23:06:17.888932+0000","last_unstale":"2023-09-29T23:06:18.286862+0000","last_undegraded":"2023-09-29T23:06:18.286862+0000","last_fullsized":"2023-09-29T23:06:18.286862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280276+0000","last_change":"2023-09-29T23:06:18.271028+0000","last_active":"2023-09-29T23:06:18.280276+0000","last_peered":"2023-09-29T23:06:18.280276+0000","last_clean":"2023-09-29T23:06:18.280276+0000","last_became_active":"2023-09-29T23:06:18.270012+0000","last_became_peered":"2023-09-29T23:06:18.270012+0000","last_unstale":"2023-09-29T23:06:18.280276+0000","last_undegraded":"2023-09-29T23:06:18.280276+0000","last_fullsized":"2023-09-29T23:06:18.280276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.293088+0000","last_change":"2023-09-29T23:06:17.889224+0000","last_active":"2023-09-29T23:06:18.293088+0000","last_peered":"2023-09-29T23:06:18.293088+0000","last_clean":"2023-09-29T23:06:18.293088+0000","last_became_active":"2023-09-29T23:06:17.888941+0000","last_became_peered":"2023-09-29T23:06:17.888941+0000","last_unstale":"2023-09-29T23:06:18.293088+0000","last_undegraded":"2023-09-29T23:06:18.293088+0000","last_fullsized":"2023-09-29T23:06:18.293088+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:06:09.687876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274555+0000","last_change":"2023-09-29T23:06:18.270435+0000","last_active":"2023-09-29T23:06:18.274555+0000","last_peered":"2023-09-29T23:06:18.274555+0000","last_clean":"2023-09-29T23:06:18.274555+0000","last_became_active":"2023-09-29T23:06:18.268172+0000","last_became_peered":"2023-09-29T23:06:18.268172+0000","last_unstale":"2023-09-29T23:06:18.274555+0000","last_undegraded":"2023-09-29T23:06:18.274555+0000","last_fullsized":"2023-09-29T23:06:18.274555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:45.768259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289061+0000","last_change":"2023-09-29T23:06:17.893695+0000","last_active":"2023-09-29T23:06:18.289061+0000","last_peered":"2023-09-29T23:06:18.289061+0000","last_clean":"2023-09-29T23:06:18.289061+0000","last_became_active":"2023-09-29T23:06:17.893381+0000","last_became_peered":"2023-09-29T23:06:17.893381+0000","last_unstale":"2023-09-29T23:06:18.289061+0000","last_undegraded":"2023-09-29T23:06:18.289061+0000","last_fullsized":"2023-09-29T23:06:18.289061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292208+0000","last_change":"2023-09-29T23:06:18.287485+0000","last_active":"2023-09-29T23:06:18.292208+0000","last_peered":"2023-09-29T23:06:18.292208+0000","last_clean":"2023-09-29T23:06:18.292208+0000","last_became_active":"2023-09-29T23:06:18.285220+0000","last_became_peered":"2023-09-29T23:06:18.285220+0000","last_unstale":"2023-09-29T23:06:18.292208+0000","last_undegraded":"2023-09-29T23:06:18.292208+0000","last_fullsized":"2023-09-29T23:06:18.292208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260745+0000","last_change":"2023-09-29T23:06:17.888841+0000","last_active":"2023-09-29T23:06:18.260745+0000","last_peered":"2023-09-29T23:06:18.260745+0000","last_clean":"2023-09-29T23:06:18.260745+0000","last_became_active":"2023-09-29T23:06:17.888515+0000","last_became_peered":"2023-09-29T23:06:17.888515+0000","last_unstale":"2023-09-29T23:06:18.260745+0000","last_undegraded":"2023-09-29T23:06:18.260745+0000","last_fullsized":"2023-09-29T23:06:18.260745+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287868+0000","last_change":"2023-09-29T23:06:18.285208+0000","last_active":"2023-09-29T23:06:18.287868+0000","last_peered":"2023-09-29T23:06:18.287868+0000","last_clean":"2023-09-29T23:06:18.287868+0000","last_became_active":"2023-09-29T23:06:18.284879+0000","last_became_peered":"2023-09-29T23:06:18.284879+0000","last_unstale":"2023-09-29T23:06:18.287868+0000","last_undegraded":"2023-09-29T23:06:18.287868+0000","last_fullsized":"2023-09-29T23:06:18.287868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260580+0000","last_change":"2023-09-29T23:06:17.890315+0000","last_active":"2023-09-29T23:06:18.260580+0000","last_peered":"2023-09-29T23:06:18.260580+0000","last_clean":"2023-09-29T23:06:18.260580+0000","last_became_active":"2023-09-29T23:06:17.890017+0000","last_became_peered":"2023-09-29T23:06:17.890017+0000","last_unstale":"2023-09-29T23:06:18.260580+0000","last_undegraded":"2023-09-29T23:06:18.260580+0000","last_fullsized":"2023-09-29T23:06:18.260580+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:55:35.860921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288529+0000","last_change":"2023-09-29T23:06:18.285771+0000","last_active":"2023-09-29T23:06:18.288529+0000","last_peered":"2023-09-29T23:06:18.288529+0000","last_clean":"2023-09-29T23:06:18.288529+0000","last_became_active":"2023-09-29T23:06:18.285493+0000","last_became_peered":"2023-09-29T23:06:18.285493+0000","last_unstale":"2023-09-29T23:06:18.288529+0000","last_undegraded":"2023-09-29T23:06:18.288529+0000","last_fullsized":"2023-09-29T23:06:18.288529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282000+0000","last_change":"2023-09-29T23:06:17.891835+0000","last_active":"2023-09-29T23:06:18.282000+0000","last_peered":"2023-09-29T23:06:18.282000+0000","last_clean":"2023-09-29T23:06:18.282000+0000","last_became_active":"2023-09-29T23:06:17.891563+0000","last_became_peered":"2023-09-29T23:06:17.891563+0000","last_unstale":"2023-09-29T23:06:18.282000+0000","last_undegraded":"2023-09-29T23:06:18.282000+0000","last_fullsized":"2023-09-29T23:06:18.282000+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:22.597416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292423+0000","last_change":"2023-09-29T23:06:18.289343+0000","last_active":"2023-09-29T23:06:18.292423+0000","last_peered":"2023-09-29T23:06:18.292423+0000","last_clean":"2023-09-29T23:06:18.292423+0000","last_became_active":"2023-09-29T23:06:18.289072+0000","last_became_peered":"2023-09-29T23:06:18.289072+0000","last_unstale":"2023-09-29T23:06:18.292423+0000","last_undegraded":"2023-09-29T23:06:18.292423+0000","last_fullsized":"2023-09-29T23:06:18.292423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:27:12.781947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282222+0000","last_change":"2023-09-29T23:06:17.891844+0000","last_active":"2023-09-29T23:06:18.282222+0000","last_peered":"2023-09-29T23:06:18.282222+0000","last_clean":"2023-09-29T23:06:18.282222+0000","last_became_active":"2023-09-29T23:06:17.891582+0000","last_became_peered":"2023-09-29T23:06:17.891582+0000","last_unstale":"2023-09-29T23:06:18.282222+0000","last_undegraded":"2023-09-29T23:06:18.282222+0000","last_fullsized":"2023-09-29T23:06:18.282222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:59.037943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287898+0000","last_change":"2023-09-29T23:06:18.285944+0000","last_active":"2023-09-29T23:06:18.287898+0000","last_peered":"2023-09-29T23:06:18.287898+0000","last_clean":"2023-09-29T23:06:18.287898+0000","last_became_active":"2023-09-29T23:06:18.285661+0000","last_became_peered":"2023-09-29T23:06:18.285661+0000","last_unstale":"2023-09-29T23:06:18.287898+0000","last_undegraded":"2023-09-29T23:06:18.287898+0000","last_fullsized":"2023-09-29T23:06:18.287898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:20:25.540032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.275819+0000","last_change":"2023-09-29T23:06:17.889165+0000","last_active":"2023-09-29T23:06:18.275819+0000","last_peered":"2023-09-29T23:06:18.275819+0000","last_clean":"2023-09-29T23:06:18.275819+0000","last_became_active":"2023-09-29T23:06:17.888842+0000","last_became_peered":"2023-09-29T23:06:17.888842+0000","last_unstale":"2023-09-29T23:06:18.275819+0000","last_undegraded":"2023-09-29T23:06:18.275819+0000","last_fullsized":"2023-09-29T23:06:18.275819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:35.202255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287772+0000","last_change":"2023-09-29T23:06:18.277402+0000","last_active":"2023-09-29T23:06:18.287772+0000","last_peered":"2023-09-29T23:06:18.287772+0000","last_clean":"2023-09-29T23:06:18.287772+0000","last_became_active":"2023-09-29T23:06:18.277159+0000","last_became_peered":"2023-09-29T23:06:18.277159+0000","last_unstale":"2023-09-29T23:06:18.287772+0000","last_undegraded":"2023-09-29T23:06:18.287772+0000","last_fullsized":"2023-09-29T23:06:18.287772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273008+0000","last_change":"2023-09-29T23:06:17.890237+0000","last_active":"2023-09-29T23:06:18.273008+0000","last_peered":"2023-09-29T23:06:18.273008+0000","last_clean":"2023-09-29T23:06:18.273008+0000","last_became_active":"2023-09-29T23:06:17.889971+0000","last_became_peered":"2023-09-29T23:06:17.889971+0000","last_unstale":"2023-09-29T23:06:18.273008+0000","last_undegraded":"2023-09-29T23:06:18.273008+0000","last_fullsized":"2023-09-29T23:06:18.273008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:57.168709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282466+0000","last_change":"2023-09-29T23:06:18.276262+0000","last_active":"2023-09-29T23:06:18.282466+0000","last_peered":"2023-09-29T23:06:18.282466+0000","last_clean":"2023-09-29T23:06:18.282466+0000","last_became_active":"2023-09-29T23:06:18.276040+0000","last_became_peered":"2023-09-29T23:06:18.276040+0000","last_unstale":"2023-09-29T23:06:18.282466+0000","last_undegraded":"2023-09-29T23:06:18.282466+0000","last_fullsized":"2023-09-29T23:06:18.282466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260885+0000","last_change":"2023-09-29T23:06:17.888828+0000","last_active":"2023-09-29T23:06:18.260885+0000","last_peered":"2023-09-29T23:06:18.260885+0000","last_clean":"2023-09-29T23:06:18.260885+0000","last_became_active":"2023-09-29T23:06:17.888525+0000","last_became_peered":"2023-09-29T23:06:17.888525+0000","last_unstale":"2023-09-29T23:06:18.260885+0000","last_undegraded":"2023-09-29T23:06:18.260885+0000","last_fullsized":"2023-09-29T23:06:18.260885+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:18:04.884961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285419+0000","last_change":"2023-09-29T23:06:18.283530+0000","last_active":"2023-09-29T23:06:18.285419+0000","last_peered":"2023-09-29T23:06:18.285419+0000","last_clean":"2023-09-29T23:06:18.285419+0000","last_became_active":"2023-09-29T23:06:18.281248+0000","last_became_peered":"2023-09-29T23:06:18.281248+0000","last_unstale":"2023-09-29T23:06:18.285419+0000","last_undegraded":"2023-09-29T23:06:18.285419+0000","last_fullsized":"2023-09-29T23:06:18.285419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:23.599215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289625+0000","last_change":"2023-09-29T23:06:17.894026+0000","last_active":"2023-09-29T23:06:18.289625+0000","last_peered":"2023-09-29T23:06:18.289625+0000","last_clean":"2023-09-29T23:06:18.289625+0000","last_became_active":"2023-09-29T23:06:17.893757+0000","last_became_peered":"2023-09-29T23:06:17.893757+0000","last_unstale":"2023-09-29T23:06:18.289625+0000","last_undegraded":"2023-09-29T23:06:18.289625+0000","last_fullsized":"2023-09-29T23:06:18.289625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286633+0000","last_change":"2023-09-29T23:06:18.278135+0000","last_active":"2023-09-29T23:06:18.286633+0000","last_peered":"2023-09-29T23:06:18.286633+0000","last_clean":"2023-09-29T23:06:18.286633+0000","last_became_active":"2023-09-29T23:06:18.277865+0000","last_became_peered":"2023-09-29T23:06:18.277865+0000","last_unstale":"2023-09-29T23:06:18.286633+0000","last_undegraded":"2023-09-29T23:06:18.286633+0000","last_fullsized":"2023-09-29T23:06:18.286633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260719+0000","last_change":"2023-09-29T23:06:17.891680+0000","last_active":"2023-09-29T23:06:18.260719+0000","last_peered":"2023-09-29T23:06:18.260719+0000","last_clean":"2023-09-29T23:06:18.260719+0000","last_became_active":"2023-09-29T23:06:17.891413+0000","last_became_peered":"2023-09-29T23:06:17.891413+0000","last_unstale":"2023-09-29T23:06:18.260719+0000","last_undegraded":"2023-09-29T23:06:18.260719+0000","last_fullsized":"2023-09-29T23:06:18.260719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:19.905553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269364+0000","last_change":"2023-09-29T23:06:18.268288+0000","last_active":"2023-09-29T23:06:18.269364+0000","last_peered":"2023-09-29T23:06:18.269364+0000","last_clean":"2023-09-29T23:06:18.269364+0000","last_became_active":"2023-09-29T23:06:18.268079+0000","last_became_peered":"2023-09-29T23:06:18.268079+0000","last_unstale":"2023-09-29T23:06:18.269364+0000","last_undegraded":"2023-09-29T23:06:18.269364+0000","last_fullsized":"2023-09-29T23:06:18.269364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274260+0000","last_change":"2023-09-29T23:06:17.892193+0000","last_active":"2023-09-29T23:06:18.274260+0000","last_peered":"2023-09-29T23:06:18.274260+0000","last_clean":"2023-09-29T23:06:18.274260+0000","last_became_active":"2023-09-29T23:06:17.891898+0000","last_became_peered":"2023-09-29T23:06:17.891898+0000","last_unstale":"2023-09-29T23:06:18.274260+0000","last_undegraded":"2023-09-29T23:06:18.274260+0000","last_fullsized":"2023-09-29T23:06:18.274260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.276612+0000","last_change":"2023-09-29T23:06:18.271513+0000","last_active":"2023-09-29T23:06:18.276612+0000","last_peered":"2023-09-29T23:06:18.276612+0000","last_clean":"2023-09-29T23:06:18.276612+0000","last_became_active":"2023-09-29T23:06:18.271257+0000","last_became_peered":"2023-09-29T23:06:18.271257+0000","last_unstale":"2023-09-29T23:06:18.276612+0000","last_undegraded":"2023-09-29T23:06:18.276612+0000","last_fullsized":"2023-09-29T23:06:18.276612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:23.301482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273504+0000","last_change":"2023-09-29T23:06:17.893616+0000","last_active":"2023-09-29T23:06:18.273504+0000","last_peered":"2023-09-29T23:06:18.273504+0000","last_clean":"2023-09-29T23:06:18.273504+0000","last_became_active":"2023-09-29T23:06:17.893285+0000","last_became_peered":"2023-09-29T23:06:17.893285+0000","last_unstale":"2023-09-29T23:06:18.273504+0000","last_undegraded":"2023-09-29T23:06:18.273504+0000","last_fullsized":"2023-09-29T23:06:18.273504+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283690+0000","last_change":"2023-09-29T23:06:18.277473+0000","last_active":"2023-09-29T23:06:18.283690+0000","last_peered":"2023-09-29T23:06:18.283690+0000","last_clean":"2023-09-29T23:06:18.283690+0000","last_became_active":"2023-09-29T23:06:18.276953+0000","last_became_peered":"2023-09-29T23:06:18.276953+0000","last_unstale":"2023-09-29T23:06:18.283690+0000","last_undegraded":"2023-09-29T23:06:18.283690+0000","last_fullsized":"2023-09-29T23:06:18.283690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264133+0000","last_change":"2023-09-29T23:06:13.639723+0000","last_active":"2023-09-29T23:06:18.264133+0000","last_peered":"2023-09-29T23:06:18.264133+0000","last_clean":"2023-09-29T23:06:18.264133+0000","last_became_active":"2023-09-29T23:06:13.639406+0000","last_became_peered":"2023-09-29T23:06:13.639406+0000","last_unstale":"2023-09-29T23:06:18.264133+0000","last_undegraded":"2023-09-29T23:06:18.264133+0000","last_fullsized":"2023-09-29T23:06:18.264133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:07:23.007465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261412+0000","last_change":"2023-09-29T23:06:17.894258+0000","last_active":"2023-09-29T23:06:18.261412+0000","last_peered":"2023-09-29T23:06:18.261412+0000","last_clean":"2023-09-29T23:06:18.261412+0000","last_became_active":"2023-09-29T23:06:17.893935+0000","last_became_peered":"2023-09-29T23:06:17.893935+0000","last_unstale":"2023-09-29T23:06:18.261412+0000","last_undegraded":"2023-09-29T23:06:18.261412+0000","last_fullsized":"2023-09-29T23:06:18.261412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:30:41.674282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280296+0000","last_change":"2023-09-29T23:06:18.271186+0000","last_active":"2023-09-29T23:06:18.280296+0000","last_peered":"2023-09-29T23:06:18.280296+0000","last_clean":"2023-09-29T23:06:18.280296+0000","last_became_active":"2023-09-29T23:06:18.270718+0000","last_became_peered":"2023-09-29T23:06:18.270718+0000","last_unstale":"2023-09-29T23:06:18.280296+0000","last_undegraded":"2023-09-29T23:06:18.280296+0000","last_fullsized":"2023-09-29T23:06:18.280296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:58.170509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260686+0000","last_change":"2023-09-29T23:06:13.380282+0000","last_active":"2023-09-29T23:06:18.260686+0000","last_peered":"2023-09-29T23:06:18.260686+0000","last_clean":"2023-09-29T23:06:18.260686+0000","last_became_active":"2023-09-29T23:06:13.379948+0000","last_became_peered":"2023-09-29T23:06:13.379948+0000","last_unstale":"2023-09-29T23:06:18.260686+0000","last_undegraded":"2023-09-29T23:06:18.260686+0000","last_fullsized":"2023-09-29T23:06:18.260686+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:02.759420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261279+0000","last_change":"2023-09-29T23:06:17.884199+0000","last_active":"2023-09-29T23:06:18.261279+0000","last_peered":"2023-09-29T23:06:18.261279+0000","last_clean":"2023-09-29T23:06:18.261279+0000","last_became_active":"2023-09-29T23:06:17.883932+0000","last_became_peered":"2023-09-29T23:06:17.883932+0000","last_unstale":"2023-09-29T23:06:18.261279+0000","last_undegraded":"2023-09-29T23:06:18.261279+0000","last_fullsized":"2023-09-29T23:06:18.261279+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:39.182159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283703+0000","last_change":"2023-09-29T23:06:18.277498+0000","last_active":"2023-09-29T23:06:18.283703+0000","last_peered":"2023-09-29T23:06:18.283703+0000","last_clean":"2023-09-29T23:06:18.283703+0000","last_became_active":"2023-09-29T23:06:18.276885+0000","last_became_peered":"2023-09-29T23:06:18.276885+0000","last_unstale":"2023-09-29T23:06:18.283703+0000","last_undegraded":"2023-09-29T23:06:18.283703+0000","last_fullsized":"2023-09-29T23:06:18.283703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.259675+0000","last_change":"2023-09-29T23:06:13.639753+0000","last_active":"2023-09-29T23:06:18.259675+0000","last_peered":"2023-09-29T23:06:18.259675+0000","last_clean":"2023-09-29T23:06:18.259675+0000","last_became_active":"2023-09-29T23:06:13.639436+0000","last_became_peered":"2023-09-29T23:06:13.639436+0000","last_unstale":"2023-09-29T23:06:18.259675+0000","last_undegraded":"2023-09-29T23:06:18.259675+0000","last_fullsized":"2023-09-29T23:06:18.259675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:09:57.060849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261100+0000","last_change":"2023-09-29T23:06:17.885616+0000","last_active":"2023-09-29T23:06:18.261100+0000","last_peered":"2023-09-29T23:06:18.261100+0000","last_clean":"2023-09-29T23:06:18.261100+0000","last_became_active":"2023-09-29T23:06:17.885342+0000","last_became_peered":"2023-09-29T23:06:17.885342+0000","last_unstale":"2023-09-29T23:06:18.261100+0000","last_undegraded":"2023-09-29T23:06:18.261100+0000","last_fullsized":"2023-09-29T23:06:18.261100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280779+0000","last_change":"2023-09-29T23:06:18.277223+0000","last_active":"2023-09-29T23:06:18.280779+0000","last_peered":"2023-09-29T23:06:18.280779+0000","last_clean":"2023-09-29T23:06:18.280779+0000","last_became_active":"2023-09-29T23:06:18.276548+0000","last_became_peered":"2023-09-29T23:06:18.276548+0000","last_unstale":"2023-09-29T23:06:18.280779+0000","last_undegraded":"2023-09-29T23:06:18.280779+0000","last_fullsized":"2023-09-29T23:06:18.280779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261249+0000","last_change":"2023-09-29T23:06:13.380967+0000","last_active":"2023-09-29T23:06:18.261249+0000","last_peered":"2023-09-29T23:06:18.261249+0000","last_clean":"2023-09-29T23:06:18.261249+0000","last_became_active":"2023-09-29T23:06:13.379890+0000","last_became_peered":"2023-09-29T23:06:13.379890+0000","last_unstale":"2023-09-29T23:06:18.261249+0000","last_undegraded":"2023-09-29T23:06:18.261249+0000","last_fullsized":"2023-09-29T23:06:18.261249+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:46.738824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260986+0000","last_change":"2023-09-29T23:06:09.717533+0000","last_active":"2023-09-29T23:06:18.260986+0000","last_peered":"2023-09-29T23:06:18.260986+0000","last_clean":"2023-09-29T23:06:18.260986+0000","last_became_active":"2023-09-29T23:06:09.716536+0000","last_became_peered":"2023-09-29T23:06:09.716536+0000","last_unstale":"2023-09-29T23:06:18.260986+0000","last_undegraded":"2023-09-29T23:06:18.260986+0000","last_fullsized":"2023-09-29T23:06:18.260986+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_clean_scrub_stamp":"2023-09-29T23:06:08.290533+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:44:31.128783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274816+0000","last_change":"2023-09-29T23:06:17.892420+0000","last_active":"2023-09-29T23:06:18.274816+0000","last_peered":"2023-09-29T23:06:18.274816+0000","last_clean":"2023-09-29T23:06:18.274816+0000","last_became_active":"2023-09-29T23:06:17.892121+0000","last_became_peered":"2023-09-29T23:06:17.892121+0000","last_unstale":"2023-09-29T23:06:18.274816+0000","last_undegraded":"2023-09-29T23:06:18.274816+0000","last_fullsized":"2023-09-29T23:06:18.274816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283140+0000","last_change":"2023-09-29T23:06:18.281364+0000","last_active":"2023-09-29T23:06:18.283140+0000","last_peered":"2023-09-29T23:06:18.283140+0000","last_clean":"2023-09-29T23:06:18.283140+0000","last_became_active":"2023-09-29T23:06:18.280821+0000","last_became_peered":"2023-09-29T23:06:18.280821+0000","last_unstale":"2023-09-29T23:06:18.283140+0000","last_undegraded":"2023-09-29T23:06:18.283140+0000","last_fullsized":"2023-09-29T23:06:18.283140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260488+0000","last_change":"2023-09-29T23:06:13.380209+0000","last_active":"2023-09-29T23:06:18.260488+0000","last_peered":"2023-09-29T23:06:18.260488+0000","last_clean":"2023-09-29T23:06:18.260488+0000","last_became_active":"2023-09-29T23:06:13.379845+0000","last_became_peered":"2023-09-29T23:06:13.379845+0000","last_unstale":"2023-09-29T23:06:18.260488+0000","last_undegraded":"2023-09-29T23:06:18.260488+0000","last_fullsized":"2023-09-29T23:06:18.260488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261391+0000","last_change":"2023-09-29T23:06:17.891971+0000","last_active":"2023-09-29T23:06:18.261391+0000","last_peered":"2023-09-29T23:06:18.261391+0000","last_clean":"2023-09-29T23:06:18.261391+0000","last_became_active":"2023-09-29T23:06:17.891695+0000","last_became_peered":"2023-09-29T23:06:17.891695+0000","last_unstale":"2023-09-29T23:06:18.261391+0000","last_undegraded":"2023-09-29T23:06:18.261391+0000","last_fullsized":"2023-09-29T23:06:18.261391+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288649+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288351+0000","last_became_peered":"2023-09-29T23:06:18.288351+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261483+0000","last_change":"2023-09-29T23:06:13.380613+0000","last_active":"2023-09-29T23:06:18.261483+0000","last_peered":"2023-09-29T23:06:18.261483+0000","last_clean":"2023-09-29T23:06:18.261483+0000","last_became_active":"2023-09-29T23:06:13.380347+0000","last_became_peered":"2023-09-29T23:06:13.380347+0000","last_unstale":"2023-09-29T23:06:18.261483+0000","last_undegraded":"2023-09-29T23:06:18.261483+0000","last_fullsized":"2023-09-29T23:06:18.261483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:35.568211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264019+0000","last_change":"2023-09-29T23:06:17.885489+0000","last_active":"2023-09-29T23:06:18.264019+0000","last_peered":"2023-09-29T23:06:18.264019+0000","last_clean":"2023-09-29T23:06:18.264019+0000","last_became_active":"2023-09-29T23:06:17.884966+0000","last_became_peered":"2023-09-29T23:06:17.884966+0000","last_unstale":"2023-09-29T23:06:18.264019+0000","last_undegraded":"2023-09-29T23:06:18.264019+0000","last_fullsized":"2023-09-29T23:06:18.264019+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286897+0000","last_change":"2023-09-29T23:06:18.282836+0000","last_active":"2023-09-29T23:06:18.286897+0000","last_peered":"2023-09-29T23:06:18.286897+0000","last_clean":"2023-09-29T23:06:18.286897+0000","last_became_active":"2023-09-29T23:06:18.282215+0000","last_became_peered":"2023-09-29T23:06:18.282215+0000","last_unstale":"2023-09-29T23:06:18.286897+0000","last_undegraded":"2023-09-29T23:06:18.286897+0000","last_fullsized":"2023-09-29T23:06:18.286897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288992+0000","last_change":"2023-09-29T23:06:17.894025+0000","last_active":"2023-09-29T23:06:18.288992+0000","last_peered":"2023-09-29T23:06:18.288992+0000","last_clean":"2023-09-29T23:06:18.288992+0000","last_became_active":"2023-09-29T23:06:17.893701+0000","last_became_peered":"2023-09-29T23:06:17.893701+0000","last_unstale":"2023-09-29T23:06:18.288992+0000","last_undegraded":"2023-09-29T23:06:18.288992+0000","last_fullsized":"2023-09-29T23:06:18.288992+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.270574+0000","last_change":"2023-09-29T23:06:13.379718+0000","last_active":"2023-09-29T23:06:18.270574+0000","last_peered":"2023-09-29T23:06:18.270574+0000","last_clean":"2023-09-29T23:06:18.270574+0000","last_became_active":"2023-09-29T23:06:13.379450+0000","last_became_peered":"2023-09-29T23:06:13.379450+0000","last_unstale":"2023-09-29T23:06:18.270574+0000","last_undegraded":"2023-09-29T23:06:18.270574+0000","last_fullsized":"2023-09-29T23:06:18.270574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286470+0000","last_change":"2023-09-29T23:06:18.283988+0000","last_active":"2023-09-29T23:06:18.286470+0000","last_peered":"2023-09-29T23:06:18.286470+0000","last_clean":"2023-09-29T23:06:18.286470+0000","last_became_active":"2023-09-29T23:06:18.283448+0000","last_became_peered":"2023-09-29T23:06:18.283448+0000","last_unstale":"2023-09-29T23:06:18.286470+0000","last_undegraded":"2023-09-29T23:06:18.286470+0000","last_fullsized":"2023-09-29T23:06:18.286470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260973+0000","last_change":"2023-09-29T23:06:13.382829+0000","last_active":"2023-09-29T23:06:18.260973+0000","last_peered":"2023-09-29T23:06:18.260973+0000","last_clean":"2023-09-29T23:06:18.260973+0000","last_became_active":"2023-09-29T23:06:13.382590+0000","last_became_peered":"2023-09-29T23:06:13.382590+0000","last_unstale":"2023-09-29T23:06:18.260973+0000","last_undegraded":"2023-09-29T23:06:18.260973+0000","last_fullsized":"2023-09-29T23:06:18.260973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:06.822425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-09-29T23:06:19.267536+0000","last_change":"2023-09-29T23:06:17.881636+0000","last_active":"2023-09-29T23:06:19.267536+0000","last_peered":"2023-09-29T23:06:19.267536+0000","last_clean":"2023-09-29T23:06:19.267536+0000","last_became_active":"2023-09-29T23:06:17.881343+0000","last_became_peered":"2023-09-29T23:06:17.881343+0000","last_unstale":"2023-09-29T23:06:19.267536+0000","last_undegraded":"2023-09-29T23:06:19.267536+0000","last_fullsized":"2023-09-29T23:06:19.267536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274553+0000","last_change":"2023-09-29T23:06:18.268514+0000","last_active":"2023-09-29T23:06:18.274553+0000","last_peered":"2023-09-29T23:06:18.274553+0000","last_clean":"2023-09-29T23:06:18.274553+0000","last_became_active":"2023-09-29T23:06:18.268089+0000","last_became_peered":"2023-09-29T23:06:18.268089+0000","last_unstale":"2023-09-29T23:06:18.274553+0000","last_undegraded":"2023-09-29T23:06:18.274553+0000","last_fullsized":"2023-09-29T23:06:18.274553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:22.930009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288913+0000","last_change":"2023-09-29T23:06:17.892888+0000","last_active":"2023-09-29T23:06:18.288913+0000","last_peered":"2023-09-29T23:06:18.288913+0000","last_clean":"2023-09-29T23:06:18.288913+0000","last_became_active":"2023-09-29T23:06:17.892666+0000","last_became_peered":"2023-09-29T23:06:17.892666+0000","last_unstale":"2023-09-29T23:06:18.288913+0000","last_undegraded":"2023-09-29T23:06:18.288913+0000","last_fullsized":"2023-09-29T23:06:18.288913+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285341+0000","last_change":"2023-09-29T23:06:18.282028+0000","last_active":"2023-09-29T23:06:18.285341+0000","last_peered":"2023-09-29T23:06:18.285341+0000","last_clean":"2023-09-29T23:06:18.285341+0000","last_became_active":"2023-09-29T23:06:18.281546+0000","last_became_peered":"2023-09-29T23:06:18.281546+0000","last_unstale":"2023-09-29T23:06:18.285341+0000","last_undegraded":"2023-09-29T23:06:18.285341+0000","last_fullsized":"2023-09-29T23:06:18.285341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:20.907352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285659+0000","last_change":"2023-09-29T23:06:17.891510+0000","last_active":"2023-09-29T23:06:18.285659+0000","last_peered":"2023-09-29T23:06:18.285659+0000","last_clean":"2023-09-29T23:06:18.285659+0000","last_became_active":"2023-09-29T23:06:17.891217+0000","last_became_peered":"2023-09-29T23:06:17.891217+0000","last_unstale":"2023-09-29T23:06:18.285659+0000","last_undegraded":"2023-09-29T23:06:18.285659+0000","last_fullsized":"2023-09-29T23:06:18.285659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285386+0000","last_change":"2023-09-29T23:06:18.283220+0000","last_active":"2023-09-29T23:06:18.285386+0000","last_peered":"2023-09-29T23:06:18.285386+0000","last_clean":"2023-09-29T23:06:18.285386+0000","last_became_active":"2023-09-29T23:06:18.281619+0000","last_became_peered":"2023-09-29T23:06:18.281619+0000","last_unstale":"2023-09-29T23:06:18.285386+0000","last_undegraded":"2023-09-29T23:06:18.285386+0000","last_fullsized":"2023-09-29T23:06:18.285386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.039742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.272636+0000","last_change":"2023-09-29T23:06:17.887773+0000","last_active":"2023-09-29T23:06:18.272636+0000","last_peered":"2023-09-29T23:06:18.272636+0000","last_clean":"2023-09-29T23:06:18.272636+0000","last_became_active":"2023-09-29T23:06:17.887578+0000","last_became_peered":"2023-09-29T23:06:17.887578+0000","last_unstale":"2023-09-29T23:06:18.272636+0000","last_undegraded":"2023-09-29T23:06:18.272636+0000","last_fullsized":"2023-09-29T23:06:18.272636+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:04:22.787417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288175+0000","last_change":"2023-09-29T23:06:18.287213+0000","last_active":"2023-09-29T23:06:18.288175+0000","last_peered":"2023-09-29T23:06:18.288175+0000","last_clean":"2023-09-29T23:06:18.288175+0000","last_became_active":"2023-09-29T23:06:18.286861+0000","last_became_peered":"2023-09-29T23:06:18.286861+0000","last_unstale":"2023-09-29T23:06:18.288175+0000","last_undegraded":"2023-09-29T23:06:18.288175+0000","last_fullsized":"2023-09-29T23:06:18.288175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:44:31.777462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286577+0000","last_change":"2023-09-29T23:06:17.893529+0000","last_active":"2023-09-29T23:06:18.286577+0000","last_peered":"2023-09-29T23:06:18.286577+0000","last_clean":"2023-09-29T23:06:18.286577+0000","last_became_active":"2023-09-29T23:06:17.893269+0000","last_became_peered":"2023-09-29T23:06:17.893269+0000","last_unstale":"2023-09-29T23:06:18.286577+0000","last_undegraded":"2023-09-29T23:06:18.286577+0000","last_fullsized":"2023-09-29T23:06:18.286577+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288800+0000","last_change":"2023-09-29T23:06:18.286214+0000","last_active":"2023-09-29T23:06:18.288800+0000","last_peered":"2023-09-29T23:06:18.288800+0000","last_clean":"2023-09-29T23:06:18.288800+0000","last_became_active":"2023-09-29T23:06:18.285773+0000","last_became_peered":"2023-09-29T23:06:18.285773+0000","last_unstale":"2023-09-29T23:06:18.288800+0000","last_undegraded":"2023-09-29T23:06:18.288800+0000","last_fullsized":"2023-09-29T23:06:18.288800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278612+0000","last_change":"2023-09-29T23:06:17.892342+0000","last_active":"2023-09-29T23:06:18.278612+0000","last_peered":"2023-09-29T23:06:18.278612+0000","last_clean":"2023-09-29T23:06:18.278612+0000","last_became_active":"2023-09-29T23:06:17.891454+0000","last_became_peered":"2023-09-29T23:06:17.891454+0000","last_unstale":"2023-09-29T23:06:18.278612+0000","last_undegraded":"2023-09-29T23:06:18.278612+0000","last_fullsized":"2023-09-29T23:06:18.278612+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283064+0000","last_change":"2023-09-29T23:06:18.280840+0000","last_active":"2023-09-29T23:06:18.283064+0000","last_peered":"2023-09-29T23:06:18.283064+0000","last_clean":"2023-09-29T23:06:18.283064+0000","last_became_active":"2023-09-29T23:06:18.280656+0000","last_became_peered":"2023-09-29T23:06:18.280656+0000","last_unstale":"2023-09-29T23:06:18.283064+0000","last_undegraded":"2023-09-29T23:06:18.283064+0000","last_fullsized":"2023-09-29T23:06:18.283064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:13.688293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273870+0000","last_change":"2023-09-29T23:06:18.272838+0000","last_active":"2023-09-29T23:06:18.273870+0000","last_peered":"2023-09-29T23:06:18.273870+0000","last_clean":"2023-09-29T23:06:18.273870+0000","last_became_active":"2023-09-29T23:06:18.272444+0000","last_became_peered":"2023-09-29T23:06:18.272444+0000","last_unstale":"2023-09-29T23:06:18.273870+0000","last_undegraded":"2023-09-29T23:06:18.273870+0000","last_fullsized":"2023-09-29T23:06:18.273870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288812+0000","last_change":"2023-09-29T23:06:18.286345+0000","last_active":"2023-09-29T23:06:18.288812+0000","last_peered":"2023-09-29T23:06:18.288812+0000","last_clean":"2023-09-29T23:06:18.288812+0000","last_became_active":"2023-09-29T23:06:18.285985+0000","last_became_peered":"2023-09-29T23:06:18.285985+0000","last_unstale":"2023-09-29T23:06:18.288812+0000","last_undegraded":"2023-09-29T23:06:18.288812+0000","last_fullsized":"2023-09-29T23:06:18.288812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280311+0000","last_change":"2023-09-29T23:06:18.271411+0000","last_active":"2023-09-29T23:06:18.280311+0000","last_peered":"2023-09-29T23:06:18.280311+0000","last_clean":"2023-09-29T23:06:18.280311+0000","last_became_active":"2023-09-29T23:06:18.270318+0000","last_became_peered":"2023-09-29T23:06:18.270318+0000","last_unstale":"2023-09-29T23:06:18.280311+0000","last_undegraded":"2023-09-29T23:06:18.280311+0000","last_fullsized":"2023-09-29T23:06:18.280311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292425+0000","last_change":"2023-09-29T23:06:18.289646+0000","last_active":"2023-09-29T23:06:18.292425+0000","last_peered":"2023-09-29T23:06:18.292425+0000","last_clean":"2023-09-29T23:06:18.292425+0000","last_became_active":"2023-09-29T23:06:18.289304+0000","last_became_peered":"2023-09-29T23:06:18.289304+0000","last_unstale":"2023-09-29T23:06:18.292425+0000","last_undegraded":"2023-09-29T23:06:18.292425+0000","last_fullsized":"2023-09-29T23:06:18.292425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:18:01.223023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283682+0000","last_change":"2023-09-29T23:06:18.277502+0000","last_active":"2023-09-29T23:06:18.283682+0000","last_peered":"2023-09-29T23:06:18.283682+0000","last_clean":"2023-09-29T23:06:18.283682+0000","last_became_active":"2023-09-29T23:06:18.277224+0000","last_became_peered":"2023-09-29T23:06:18.277224+0000","last_unstale":"2023-09-29T23:06:18.283682+0000","last_undegraded":"2023-09-29T23:06:18.283682+0000","last_fullsized":"2023-09-29T23:06:18.283682+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":188416,"data_stored":60567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8960,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362880,"statfs":{"total":96636764160,"available":96627589120,"internally_reserved":0,"allocated":253952,"data_stored":652921,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8588,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363252,"statfs":{"total":96636764160,"available":96627970048,"internally_reserved":0,"allocated":200704,"data_stored":62035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9076,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362764,"statfs":{"total":96636764160,"available":96627470336,"internally_reserved":0,"allocated":241664,"data_stored":651317,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:06:30.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-29T23:06:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.849+0000 7f6c95c49700 1 Processor -- start 2023-09-29T23:06:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.849+0000 7f6c95c49700 1 -- start start 2023-09-29T23:06:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.849+0000 7f6c95c49700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6c9006a6f0 0x7f6c900a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.849+0000 7f6c95c49700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9006eea0 con 0x7f6c90067810 2023-09-29T23:06:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.849+0000 7f6c95c49700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9006f0d0 con 0x7f6c90067440 2023-09-29T23:06:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.849+0000 7f6c95c49700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9006a1c0 con 0x7f6c9006a6f0 2023-09-29T23:06:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8effd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6c90067810 0x7f6c90069ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44532/0 (socket says 172.21.15.121:44532) 2023-09-29T23:06:30.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8effd700 1 -- 172.21.15.121:0/2466616835 learned_addr learned my addr 172.21.15.121:0/2466616835 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8ffff700 1 --2- 172.21.15.121:0/2466616835 >> v2:172.21.15.123:3300/0 conn(0x7f6c9006a6f0 0x7f6c900a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8e7fc700 1 -- 172.21.15.121:0/2466616835 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681118236 0 0) 0x7f6c9006eea0 con 0x7f6c90067810 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8e7fc700 1 -- 172.21.15.121:0/2466616835 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c78003570 con 0x7f6c90067810 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8ffff700 1 -- 172.21.15.121:0/2466616835 >> v1:172.21.15.123:6789/0 conn(0x7f6c90067440 legacy=0x7f6c9006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8ffff700 1 -- 172.21.15.121:0/2466616835 >> v1:172.21.15.121:6789/0 conn(0x7f6c90067810 legacy=0x7f6c90069ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8ffff700 1 -- 172.21.15.121:0/2466616835 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9006dfe0 con 0x7f6c9006a6f0 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8ffff700 1 --2- 172.21.15.121:0/2466616835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c9006a6f0 0x7f6c900a5750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009e60 tx=0x7f6c88009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f23b84bc7a249e98 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8e7fc700 1 -- 172.21.15.121:0/2466616835 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800e070 con 0x7f6c9006a6f0 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8e7fc700 1 -- 172.21.15.121:0/2466616835 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8800bb60 con 0x7f6c9006a6f0 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.850+0000 7f6c8e7fc700 1 -- 172.21.15.121:0/2466616835 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88012430 con 0x7f6c9006a6f0 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.851+0000 7f6c95c49700 1 -- 172.21.15.121:0/2466616835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c9006a6f0 msgr2=0x7f6c900a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.851+0000 7f6c95c49700 1 --2- 172.21.15.121:0/2466616835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c9006a6f0 0x7f6c900a5750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009e60 tx=0x7f6c88009780 comp rx=0 tx=0).stop 2023-09-29T23:06:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.851+0000 7f6c95c49700 1 -- 172.21.15.121:0/2466616835 shutdown_connections 2023-09-29T23:06:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.851+0000 7f6c95c49700 1 --2- 172.21.15.121:0/2466616835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c9006a6f0 0x7f6c900a5750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.851+0000 7f6c95c49700 1 -- 172.21.15.121:0/2466616835 >> 172.21.15.121:0/2466616835 conn(0x7f6c90068550 msgr2=0x7f6c90068950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.851+0000 7f6c95c49700 1 -- 172.21.15.121:0/2466616835 shutdown_connections 2023-09-29T23:06:30.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.851+0000 7f6c95c49700 1 -- 172.21.15.121:0/2466616835 wait complete. 2023-09-29T23:06:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.852+0000 7f6c95c49700 1 Processor -- start 2023-09-29T23:06:30.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.852+0000 7f6c95c49700 1 -- start start 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c90067440 0x7f6c900a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c90067810 0x7f6c901509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9006a6f0 0x7f6c90150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90150390 con 0x7f6c90067810 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9009bab0 con 0x7f6c90067440 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9009bd90 con 0x7f6c9006a6f0 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c90067810 0x7f6c901509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c90067810 0x7f6c901509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58702/0 (socket says 172.21.15.121:58702) 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 -- 172.21.15.121:0/3931306990 learned_addr learned my addr 172.21.15.121:0/3931306990 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8ffff700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9006a6f0 0x7f6c90150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8f7fe700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c90067440 0x7f6c900a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 -- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9006a6f0 msgr2=0x7f6c90150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9006a6f0 0x7f6c90150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 -- 172.21.15.121:0/3931306990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c90067440 msgr2=0x7f6c900a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c90067440 0x7f6c900a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 -- 172.21.15.121:0/3931306990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88009a20 con 0x7f6c90067810 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c8effd700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c90067810 0x7f6c901509d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400c7c0 tx=0x7f6c84007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c777fe700 1 -- 172.21.15.121:0/3931306990 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84005cd0 con 0x7f6c90067810 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c777fe700 1 -- 172.21.15.121:0/3931306990 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c84005e30 con 0x7f6c90067810 2023-09-29T23:06:30.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c777fe700 1 -- 172.21.15.121:0/3931306990 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84007320 con 0x7f6c90067810 2023-09-29T23:06:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c90150580 con 0x7f6c90067810 2023-09-29T23:06:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.853+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c90151450 con 0x7f6c90067810 2023-09-29T23:06:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.854+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c900a4080 con 0x7f6c90067810 2023-09-29T23:06:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c777fe700 1 -- 172.21.15.121:0/3931306990 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6c84007690 con 0x7f6c90067810 2023-09-29T23:06:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c777fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:30.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c777fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:30.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c777fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c777fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:30.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c777fe700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c7c098780 0x7f6c7c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:30.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c8f7fe700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c7c098780 0x7f6c7c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:30.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.856+0000 7f6c777fe700 1 -- 172.21.15.121:0/3931306990 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f6c840bfee0 con 0x7f6c90067810 2023-09-29T23:06:30.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.857+0000 7f6c8f7fe700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c7c098780 0x7f6c7c09ac30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6c80003aa0 tx=0x7f6c80000bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:30.860 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:30.860+0000 7f6c777fe700 1 -- 172.21.15.121:0/3931306990 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c8408d580 con 0x7f6c90067810 2023-09-29T23:06:31.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.036+0000 7f6c95c49700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-29T23:06:31.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.036+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 --> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c90071660 con 0x7f6c7c098780 2023-09-29T23:06:31.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.043+0000 7f6c777fe700 1 -- 172.21.15.121:0/3931306990 <== mgr.4107 v2:172.21.15.121:6808/113227 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+269404 (secure 0 0 0) 0x7f6c90071660 con 0x7f6c7c098780 2023-09-29T23:06:31.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.043+0000 7f6c777fe700 20 mgrc handle_command_reply tid 0 r 0 2023-09-29T23:06:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:31.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 10 mgrc shutdown 2023-09-29T23:06:31.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c7c098780 msgr2=0x7f6c7c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c7c098780 0x7f6c7c09ac30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6c80003aa0 tx=0x7f6c80000bc0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c90067810 msgr2=0x7f6c901509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c90067810 0x7f6c901509d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400c7c0 tx=0x7f6c84007e00 comp rx=0 tx=0).stop 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 shutdown_connections 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c7c098780 0x7f6c7c09ac30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c9006a6f0 0x7f6c90150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c90067440 0x7f6c900a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 --2- 172.21.15.121:0/3931306990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c90067810 0x7f6c901509d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 >> 172.21.15.121:0/3931306990 conn(0x7f6c90068550 msgr2=0x7f6c90136490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 shutdown_connections 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.046+0000 7f6c95c49700 1 -- 172.21.15.121:0/3931306990 wait complete. 2023-09-29T23:06:31.049 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2023-09-29T23:06:31.057 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-09-29T23:06:29.488102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":35136,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":377452224,"statfs":{"total":386547056640,"available":386511077376,"internally_reserved":0,"allocated":884736,"data_stored":1426840,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.228406"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278366+0000","last_change":"2023-09-29T23:06:18.276673+0000","last_active":"2023-09-29T23:06:18.278366+0000","last_peered":"2023-09-29T23:06:18.278366+0000","last_clean":"2023-09-29T23:06:18.278366+0000","last_became_active":"2023-09-29T23:06:18.276312+0000","last_became_peered":"2023-09-29T23:06:18.276312+0000","last_unstale":"2023-09-29T23:06:18.278366+0000","last_undegraded":"2023-09-29T23:06:18.278366+0000","last_fullsized":"2023-09-29T23:06:18.278366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:40.183958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283354+0000","last_change":"2023-09-29T23:06:18.282307+0000","last_active":"2023-09-29T23:06:18.283354+0000","last_peered":"2023-09-29T23:06:18.283354+0000","last_clean":"2023-09-29T23:06:18.283354+0000","last_became_active":"2023-09-29T23:06:18.282064+0000","last_became_peered":"2023-09-29T23:06:18.282064+0000","last_unstale":"2023-09-29T23:06:18.283354+0000","last_undegraded":"2023-09-29T23:06:18.283354+0000","last_fullsized":"2023-09-29T23:06:18.283354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:10:08.336407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283157+0000","last_change":"2023-09-29T23:06:18.281514+0000","last_active":"2023-09-29T23:06:18.283157+0000","last_peered":"2023-09-29T23:06:18.283157+0000","last_clean":"2023-09-29T23:06:18.283157+0000","last_became_active":"2023-09-29T23:06:18.281144+0000","last_became_peered":"2023-09-29T23:06:18.281144+0000","last_unstale":"2023-09-29T23:06:18.283157+0000","last_undegraded":"2023-09-29T23:06:18.283157+0000","last_fullsized":"2023-09-29T23:06:18.283157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290407+0000","last_change":"2023-09-29T23:06:18.287192+0000","last_active":"2023-09-29T23:06:18.290407+0000","last_peered":"2023-09-29T23:06:18.290407+0000","last_clean":"2023-09-29T23:06:18.290407+0000","last_became_active":"2023-09-29T23:06:18.286917+0000","last_became_peered":"2023-09-29T23:06:18.286917+0000","last_unstale":"2023-09-29T23:06:18.290407+0000","last_undegraded":"2023-09-29T23:06:18.290407+0000","last_fullsized":"2023-09-29T23:06:18.290407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292437+0000","last_change":"2023-09-29T23:06:18.289803+0000","last_active":"2023-09-29T23:06:18.292437+0000","last_peered":"2023-09-29T23:06:18.292437+0000","last_clean":"2023-09-29T23:06:18.292437+0000","last_became_active":"2023-09-29T23:06:18.289379+0000","last_became_peered":"2023-09-29T23:06:18.289379+0000","last_unstale":"2023-09-29T23:06:18.292437+0000","last_undegraded":"2023-09-29T23:06:18.292437+0000","last_fullsized":"2023-09-29T23:06:18.292437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287884+0000","last_change":"2023-09-29T23:06:18.285440+0000","last_active":"2023-09-29T23:06:18.287884+0000","last_peered":"2023-09-29T23:06:18.287884+0000","last_clean":"2023-09-29T23:06:18.287884+0000","last_became_active":"2023-09-29T23:06:18.285182+0000","last_became_peered":"2023-09-29T23:06:18.285182+0000","last_unstale":"2023-09-29T23:06:18.287884+0000","last_undegraded":"2023-09-29T23:06:18.287884+0000","last_fullsized":"2023-09-29T23:06:18.287884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:04:37.136451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288766+0000","last_change":"2023-09-29T23:06:18.286007+0000","last_active":"2023-09-29T23:06:18.288766+0000","last_peered":"2023-09-29T23:06:18.288766+0000","last_clean":"2023-09-29T23:06:18.288766+0000","last_became_active":"2023-09-29T23:06:18.284765+0000","last_became_peered":"2023-09-29T23:06:18.284765+0000","last_unstale":"2023-09-29T23:06:18.288766+0000","last_undegraded":"2023-09-29T23:06:18.288766+0000","last_fullsized":"2023-09-29T23:06:18.288766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273887+0000","last_change":"2023-09-29T23:06:18.272963+0000","last_active":"2023-09-29T23:06:18.273887+0000","last_peered":"2023-09-29T23:06:18.273887+0000","last_clean":"2023-09-29T23:06:18.273887+0000","last_became_active":"2023-09-29T23:06:18.272736+0000","last_became_peered":"2023-09-29T23:06:18.272736+0000","last_unstale":"2023-09-29T23:06:18.273887+0000","last_undegraded":"2023-09-29T23:06:18.273887+0000","last_fullsized":"2023-09-29T23:06:18.273887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278347+0000","last_change":"2023-09-29T23:06:18.276308+0000","last_active":"2023-09-29T23:06:18.278347+0000","last_peered":"2023-09-29T23:06:18.278347+0000","last_clean":"2023-09-29T23:06:18.278347+0000","last_became_active":"2023-09-29T23:06:18.276056+0000","last_became_peered":"2023-09-29T23:06:18.276056+0000","last_unstale":"2023-09-29T23:06:18.278347+0000","last_undegraded":"2023-09-29T23:06:18.278347+0000","last_fullsized":"2023-09-29T23:06:18.278347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:36.204055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287088+0000","last_change":"2023-09-29T23:06:18.280125+0000","last_active":"2023-09-29T23:06:18.287088+0000","last_peered":"2023-09-29T23:06:18.287088+0000","last_clean":"2023-09-29T23:06:18.287088+0000","last_became_active":"2023-09-29T23:06:18.279830+0000","last_became_peered":"2023-09-29T23:06:18.279830+0000","last_unstale":"2023-09-29T23:06:18.287088+0000","last_undegraded":"2023-09-29T23:06:18.287088+0000","last_fullsized":"2023-09-29T23:06:18.287088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:43.753456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280206+0000","last_change":"2023-09-29T23:06:18.270148+0000","last_active":"2023-09-29T23:06:18.280206+0000","last_peered":"2023-09-29T23:06:18.280206+0000","last_clean":"2023-09-29T23:06:18.280206+0000","last_became_active":"2023-09-29T23:06:18.269859+0000","last_became_peered":"2023-09-29T23:06:18.269859+0000","last_unstale":"2023-09-29T23:06:18.280206+0000","last_undegraded":"2023-09-29T23:06:18.280206+0000","last_fullsized":"2023-09-29T23:06:18.280206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277323+0000","last_change":"2023-09-29T23:06:18.272855+0000","last_active":"2023-09-29T23:06:18.277323+0000","last_peered":"2023-09-29T23:06:18.277323+0000","last_clean":"2023-09-29T23:06:18.277323+0000","last_became_active":"2023-09-29T23:06:18.272591+0000","last_became_peered":"2023-09-29T23:06:18.272591+0000","last_unstale":"2023-09-29T23:06:18.277323+0000","last_undegraded":"2023-09-29T23:06:18.277323+0000","last_fullsized":"2023-09-29T23:06:18.277323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:35.444407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285404+0000","last_change":"2023-09-29T23:06:18.283392+0000","last_active":"2023-09-29T23:06:18.285404+0000","last_peered":"2023-09-29T23:06:18.285404+0000","last_clean":"2023-09-29T23:06:18.285404+0000","last_became_active":"2023-09-29T23:06:18.282197+0000","last_became_peered":"2023-09-29T23:06:18.282197+0000","last_unstale":"2023-09-29T23:06:18.285404+0000","last_undegraded":"2023-09-29T23:06:18.285404+0000","last_fullsized":"2023-09-29T23:06:18.285404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:33.387327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269370+0000","last_change":"2023-09-29T23:06:18.268361+0000","last_active":"2023-09-29T23:06:18.269370+0000","last_peered":"2023-09-29T23:06:18.269370+0000","last_clean":"2023-09-29T23:06:18.269370+0000","last_became_active":"2023-09-29T23:06:18.268085+0000","last_became_peered":"2023-09-29T23:06:18.268085+0000","last_unstale":"2023-09-29T23:06:18.269370+0000","last_undegraded":"2023-09-29T23:06:18.269370+0000","last_fullsized":"2023-09-29T23:06:18.269370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283344+0000","last_change":"2023-09-29T23:06:18.282177+0000","last_active":"2023-09-29T23:06:18.283344+0000","last_peered":"2023-09-29T23:06:18.283344+0000","last_clean":"2023-09-29T23:06:18.283344+0000","last_became_active":"2023-09-29T23:06:18.281841+0000","last_became_peered":"2023-09-29T23:06:18.281841+0000","last_unstale":"2023-09-29T23:06:18.283344+0000","last_undegraded":"2023-09-29T23:06:18.283344+0000","last_fullsized":"2023-09-29T23:06:18.283344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273852+0000","last_change":"2023-09-29T23:06:18.272693+0000","last_active":"2023-09-29T23:06:18.273852+0000","last_peered":"2023-09-29T23:06:18.273852+0000","last_clean":"2023-09-29T23:06:18.273852+0000","last_became_active":"2023-09-29T23:06:18.272433+0000","last_became_peered":"2023-09-29T23:06:18.272433+0000","last_unstale":"2023-09-29T23:06:18.273852+0000","last_undegraded":"2023-09-29T23:06:18.273852+0000","last_fullsized":"2023-09-29T23:06:18.273852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280760+0000","last_change":"2023-09-29T23:06:18.275372+0000","last_active":"2023-09-29T23:06:18.280760+0000","last_peered":"2023-09-29T23:06:18.280760+0000","last_clean":"2023-09-29T23:06:18.280760+0000","last_became_active":"2023-09-29T23:06:18.274861+0000","last_became_peered":"2023-09-29T23:06:18.274861+0000","last_unstale":"2023-09-29T23:06:18.280760+0000","last_undegraded":"2023-09-29T23:06:18.280760+0000","last_fullsized":"2023-09-29T23:06:18.280760+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.290310+0000","last_change":"2023-09-29T23:06:18.287098+0000","last_active":"2023-09-29T23:06:18.290310+0000","last_peered":"2023-09-29T23:06:18.290310+0000","last_clean":"2023-09-29T23:06:18.290310+0000","last_became_active":"2023-09-29T23:06:18.286847+0000","last_became_peered":"2023-09-29T23:06:18.286847+0000","last_unstale":"2023-09-29T23:06:18.290310+0000","last_undegraded":"2023-09-29T23:06:18.290310+0000","last_fullsized":"2023-09-29T23:06:18.290310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.277206+0000","last_change":"2023-09-29T23:06:18.273575+0000","last_active":"2023-09-29T23:06:18.277206+0000","last_peered":"2023-09-29T23:06:18.277206+0000","last_clean":"2023-09-29T23:06:18.277206+0000","last_became_active":"2023-09-29T23:06:18.273327+0000","last_became_peered":"2023-09-29T23:06:18.273327+0000","last_unstale":"2023-09-29T23:06:18.277206+0000","last_undegraded":"2023-09-29T23:06:18.277206+0000","last_fullsized":"2023-09-29T23:06:18.277206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:29:05.706848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280438+0000","last_change":"2023-09-29T23:06:18.271980+0000","last_active":"2023-09-29T23:06:18.280438+0000","last_peered":"2023-09-29T23:06:18.280438+0000","last_clean":"2023-09-29T23:06:18.280438+0000","last_became_active":"2023-09-29T23:06:18.271725+0000","last_became_peered":"2023-09-29T23:06:18.271725+0000","last_unstale":"2023-09-29T23:06:18.280438+0000","last_undegraded":"2023-09-29T23:06:18.280438+0000","last_fullsized":"2023-09-29T23:06:18.280438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:12:02.409965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283120+0000","last_change":"2023-09-29T23:06:18.281173+0000","last_active":"2023-09-29T23:06:18.283120+0000","last_peered":"2023-09-29T23:06:18.283120+0000","last_clean":"2023-09-29T23:06:18.283120+0000","last_became_active":"2023-09-29T23:06:18.280834+0000","last_became_peered":"2023-09-29T23:06:18.280834+0000","last_unstale":"2023-09-29T23:06:18.283120+0000","last_undegraded":"2023-09-29T23:06:18.283120+0000","last_fullsized":"2023-09-29T23:06:18.283120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:02:54.551488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280253+0000","last_change":"2023-09-29T23:06:18.270925+0000","last_active":"2023-09-29T23:06:18.280253+0000","last_peered":"2023-09-29T23:06:18.280253+0000","last_clean":"2023-09-29T23:06:18.280253+0000","last_became_active":"2023-09-29T23:06:18.269605+0000","last_became_peered":"2023-09-29T23:06:18.269605+0000","last_unstale":"2023-09-29T23:06:18.280253+0000","last_undegraded":"2023-09-29T23:06:18.280253+0000","last_fullsized":"2023-09-29T23:06:18.280253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287559+0000","last_change":"2023-09-29T23:06:18.284188+0000","last_active":"2023-09-29T23:06:18.287559+0000","last_peered":"2023-09-29T23:06:18.287559+0000","last_clean":"2023-09-29T23:06:18.287559+0000","last_became_active":"2023-09-29T23:06:18.283875+0000","last_became_peered":"2023-09-29T23:06:18.283875+0000","last_unstale":"2023-09-29T23:06:18.287559+0000","last_undegraded":"2023-09-29T23:06:18.287559+0000","last_fullsized":"2023-09-29T23:06:18.287559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:11:25.499493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283044+0000","last_change":"2023-09-29T23:06:18.280077+0000","last_active":"2023-09-29T23:06:18.283044+0000","last_peered":"2023-09-29T23:06:18.283044+0000","last_clean":"2023-09-29T23:06:18.283044+0000","last_became_active":"2023-09-29T23:06:18.279865+0000","last_became_peered":"2023-09-29T23:06:18.279865+0000","last_unstale":"2023-09-29T23:06:18.283044+0000","last_undegraded":"2023-09-29T23:06:18.283044+0000","last_fullsized":"2023-09-29T23:06:18.283044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:48.644161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280240+0000","last_change":"2023-09-29T23:06:18.270238+0000","last_active":"2023-09-29T23:06:18.280240+0000","last_peered":"2023-09-29T23:06:18.280240+0000","last_clean":"2023-09-29T23:06:18.280240+0000","last_became_active":"2023-09-29T23:06:18.269996+0000","last_became_peered":"2023-09-29T23:06:18.269996+0000","last_unstale":"2023-09-29T23:06:18.280240+0000","last_undegraded":"2023-09-29T23:06:18.280240+0000","last_fullsized":"2023-09-29T23:06:18.280240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287050+0000","last_change":"2023-09-29T23:06:18.279595+0000","last_active":"2023-09-29T23:06:18.287050+0000","last_peered":"2023-09-29T23:06:18.287050+0000","last_clean":"2023-09-29T23:06:18.287050+0000","last_became_active":"2023-09-29T23:06:18.278827+0000","last_became_peered":"2023-09-29T23:06:18.278827+0000","last_unstale":"2023-09-29T23:06:18.287050+0000","last_undegraded":"2023-09-29T23:06:18.287050+0000","last_fullsized":"2023-09-29T23:06:18.287050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:32:28.074495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288146+0000","last_change":"2023-09-29T23:06:18.287081+0000","last_active":"2023-09-29T23:06:18.288146+0000","last_peered":"2023-09-29T23:06:18.288146+0000","last_clean":"2023-09-29T23:06:18.288146+0000","last_became_active":"2023-09-29T23:06:18.286853+0000","last_became_peered":"2023-09-29T23:06:18.286853+0000","last_unstale":"2023-09-29T23:06:18.288146+0000","last_undegraded":"2023-09-29T23:06:18.288146+0000","last_fullsized":"2023-09-29T23:06:18.288146+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:28:00.791418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280662+0000","last_change":"2023-09-29T23:06:17.886342+0000","last_active":"2023-09-29T23:06:18.280662+0000","last_peered":"2023-09-29T23:06:18.280662+0000","last_clean":"2023-09-29T23:06:18.280662+0000","last_became_active":"2023-09-29T23:06:17.886089+0000","last_became_peered":"2023-09-29T23:06:17.886089+0000","last_unstale":"2023-09-29T23:06:18.280662+0000","last_undegraded":"2023-09-29T23:06:18.280662+0000","last_fullsized":"2023-09-29T23:06:18.280662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:04.152750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288614+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288301+0000","last_became_peered":"2023-09-29T23:06:18.288301+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261493+0000","last_change":"2023-09-29T23:06:17.891522+0000","last_active":"2023-09-29T23:06:18.261493+0000","last_peered":"2023-09-29T23:06:18.261493+0000","last_clean":"2023-09-29T23:06:18.261493+0000","last_became_active":"2023-09-29T23:06:17.891232+0000","last_became_peered":"2023-09-29T23:06:17.891232+0000","last_unstale":"2023-09-29T23:06:18.261493+0000","last_undegraded":"2023-09-29T23:06:18.261493+0000","last_fullsized":"2023-09-29T23:06:18.261493+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:44.766459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287070+0000","last_change":"2023-09-29T23:06:18.279774+0000","last_active":"2023-09-29T23:06:18.287070+0000","last_peered":"2023-09-29T23:06:18.287070+0000","last_clean":"2023-09-29T23:06:18.287070+0000","last_became_active":"2023-09-29T23:06:18.279014+0000","last_became_peered":"2023-09-29T23:06:18.279014+0000","last_unstale":"2023-09-29T23:06:18.287070+0000","last_undegraded":"2023-09-29T23:06:18.287070+0000","last_fullsized":"2023-09-29T23:06:18.287070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:08:08.774851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273886+0000","last_change":"2023-09-29T23:06:17.894216+0000","last_active":"2023-09-29T23:06:18.273886+0000","last_peered":"2023-09-29T23:06:18.273886+0000","last_clean":"2023-09-29T23:06:18.273886+0000","last_became_active":"2023-09-29T23:06:17.893923+0000","last_became_peered":"2023-09-29T23:06:17.893923+0000","last_unstale":"2023-09-29T23:06:18.273886+0000","last_undegraded":"2023-09-29T23:06:18.273886+0000","last_fullsized":"2023-09-29T23:06:18.273886+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:40:10.180352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282454+0000","last_change":"2023-09-29T23:06:18.276320+0000","last_active":"2023-09-29T23:06:18.282454+0000","last_peered":"2023-09-29T23:06:18.282454+0000","last_clean":"2023-09-29T23:06:18.282454+0000","last_became_active":"2023-09-29T23:06:18.276049+0000","last_became_peered":"2023-09-29T23:06:18.276049+0000","last_unstale":"2023-09-29T23:06:18.282454+0000","last_undegraded":"2023-09-29T23:06:18.282454+0000","last_fullsized":"2023-09-29T23:06:18.282454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:20:32.688280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261797+0000","last_change":"2023-09-29T23:06:17.883845+0000","last_active":"2023-09-29T23:06:18.261797+0000","last_peered":"2023-09-29T23:06:18.261797+0000","last_clean":"2023-09-29T23:06:18.261797+0000","last_became_active":"2023-09-29T23:06:17.883559+0000","last_became_peered":"2023-09-29T23:06:17.883559+0000","last_unstale":"2023-09-29T23:06:18.261797+0000","last_undegraded":"2023-09-29T23:06:18.261797+0000","last_fullsized":"2023-09-29T23:06:18.261797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286481+0000","last_change":"2023-09-29T23:06:18.284038+0000","last_active":"2023-09-29T23:06:18.286481+0000","last_peered":"2023-09-29T23:06:18.286481+0000","last_clean":"2023-09-29T23:06:18.286481+0000","last_became_active":"2023-09-29T23:06:18.283764+0000","last_became_peered":"2023-09-29T23:06:18.283764+0000","last_unstale":"2023-09-29T23:06:18.286481+0000","last_undegraded":"2023-09-29T23:06:18.286481+0000","last_fullsized":"2023-09-29T23:06:18.286481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:05:42.345774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261094+0000","last_change":"2023-09-29T23:06:17.886414+0000","last_active":"2023-09-29T23:06:18.261094+0000","last_peered":"2023-09-29T23:06:18.261094+0000","last_clean":"2023-09-29T23:06:18.261094+0000","last_became_active":"2023-09-29T23:06:17.886124+0000","last_became_peered":"2023-09-29T23:06:17.886124+0000","last_unstale":"2023-09-29T23:06:18.261094+0000","last_undegraded":"2023-09-29T23:06:18.261094+0000","last_fullsized":"2023-09-29T23:06:18.261094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:52:32.385528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288784+0000","last_change":"2023-09-29T23:06:18.286164+0000","last_active":"2023-09-29T23:06:18.288784+0000","last_peered":"2023-09-29T23:06:18.288784+0000","last_clean":"2023-09-29T23:06:18.288784+0000","last_became_active":"2023-09-29T23:06:18.285518+0000","last_became_peered":"2023-09-29T23:06:18.285518+0000","last_unstale":"2023-09-29T23:06:18.288784+0000","last_undegraded":"2023-09-29T23:06:18.288784+0000","last_fullsized":"2023-09-29T23:06:18.288784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:54:18.859428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286862+0000","last_change":"2023-09-29T23:06:17.889196+0000","last_active":"2023-09-29T23:06:18.286862+0000","last_peered":"2023-09-29T23:06:18.286862+0000","last_clean":"2023-09-29T23:06:18.286862+0000","last_became_active":"2023-09-29T23:06:17.888932+0000","last_became_peered":"2023-09-29T23:06:17.888932+0000","last_unstale":"2023-09-29T23:06:18.286862+0000","last_undegraded":"2023-09-29T23:06:18.286862+0000","last_fullsized":"2023-09-29T23:06:18.286862+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:48.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280276+0000","last_change":"2023-09-29T23:06:18.271028+0000","last_active":"2023-09-29T23:06:18.280276+0000","last_peered":"2023-09-29T23:06:18.280276+0000","last_clean":"2023-09-29T23:06:18.280276+0000","last_became_active":"2023-09-29T23:06:18.270012+0000","last_became_peered":"2023-09-29T23:06:18.270012+0000","last_unstale":"2023-09-29T23:06:18.280276+0000","last_undegraded":"2023-09-29T23:06:18.280276+0000","last_fullsized":"2023-09-29T23:06:18.280276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:21:45.656148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.293088+0000","last_change":"2023-09-29T23:06:17.889224+0000","last_active":"2023-09-29T23:06:18.293088+0000","last_peered":"2023-09-29T23:06:18.293088+0000","last_clean":"2023-09-29T23:06:18.293088+0000","last_became_active":"2023-09-29T23:06:17.888941+0000","last_became_peered":"2023-09-29T23:06:17.888941+0000","last_unstale":"2023-09-29T23:06:18.293088+0000","last_undegraded":"2023-09-29T23:06:18.293088+0000","last_fullsized":"2023-09-29T23:06:18.293088+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:06:09.687876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274555+0000","last_change":"2023-09-29T23:06:18.270435+0000","last_active":"2023-09-29T23:06:18.274555+0000","last_peered":"2023-09-29T23:06:18.274555+0000","last_clean":"2023-09-29T23:06:18.274555+0000","last_became_active":"2023-09-29T23:06:18.268172+0000","last_became_peered":"2023-09-29T23:06:18.268172+0000","last_unstale":"2023-09-29T23:06:18.274555+0000","last_undegraded":"2023-09-29T23:06:18.274555+0000","last_fullsized":"2023-09-29T23:06:18.274555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:45.768259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289061+0000","last_change":"2023-09-29T23:06:17.893695+0000","last_active":"2023-09-29T23:06:18.289061+0000","last_peered":"2023-09-29T23:06:18.289061+0000","last_clean":"2023-09-29T23:06:18.289061+0000","last_became_active":"2023-09-29T23:06:17.893381+0000","last_became_peered":"2023-09-29T23:06:17.893381+0000","last_unstale":"2023-09-29T23:06:18.289061+0000","last_undegraded":"2023-09-29T23:06:18.289061+0000","last_fullsized":"2023-09-29T23:06:18.289061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:36.802881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292208+0000","last_change":"2023-09-29T23:06:18.287485+0000","last_active":"2023-09-29T23:06:18.292208+0000","last_peered":"2023-09-29T23:06:18.292208+0000","last_clean":"2023-09-29T23:06:18.292208+0000","last_became_active":"2023-09-29T23:06:18.285220+0000","last_became_peered":"2023-09-29T23:06:18.285220+0000","last_unstale":"2023-09-29T23:06:18.292208+0000","last_undegraded":"2023-09-29T23:06:18.292208+0000","last_fullsized":"2023-09-29T23:06:18.292208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260745+0000","last_change":"2023-09-29T23:06:17.888841+0000","last_active":"2023-09-29T23:06:18.260745+0000","last_peered":"2023-09-29T23:06:18.260745+0000","last_clean":"2023-09-29T23:06:18.260745+0000","last_became_active":"2023-09-29T23:06:17.888515+0000","last_became_peered":"2023-09-29T23:06:17.888515+0000","last_unstale":"2023-09-29T23:06:18.260745+0000","last_undegraded":"2023-09-29T23:06:18.260745+0000","last_fullsized":"2023-09-29T23:06:18.260745+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:57.850925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287868+0000","last_change":"2023-09-29T23:06:18.285208+0000","last_active":"2023-09-29T23:06:18.287868+0000","last_peered":"2023-09-29T23:06:18.287868+0000","last_clean":"2023-09-29T23:06:18.287868+0000","last_became_active":"2023-09-29T23:06:18.284879+0000","last_became_peered":"2023-09-29T23:06:18.284879+0000","last_unstale":"2023-09-29T23:06:18.287868+0000","last_undegraded":"2023-09-29T23:06:18.287868+0000","last_fullsized":"2023-09-29T23:06:18.287868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260580+0000","last_change":"2023-09-29T23:06:17.890315+0000","last_active":"2023-09-29T23:06:18.260580+0000","last_peered":"2023-09-29T23:06:18.260580+0000","last_clean":"2023-09-29T23:06:18.260580+0000","last_became_active":"2023-09-29T23:06:17.890017+0000","last_became_peered":"2023-09-29T23:06:17.890017+0000","last_unstale":"2023-09-29T23:06:18.260580+0000","last_undegraded":"2023-09-29T23:06:18.260580+0000","last_fullsized":"2023-09-29T23:06:18.260580+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:55:35.860921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288529+0000","last_change":"2023-09-29T23:06:18.285771+0000","last_active":"2023-09-29T23:06:18.288529+0000","last_peered":"2023-09-29T23:06:18.288529+0000","last_clean":"2023-09-29T23:06:18.288529+0000","last_became_active":"2023-09-29T23:06:18.285493+0000","last_became_peered":"2023-09-29T23:06:18.285493+0000","last_unstale":"2023-09-29T23:06:18.288529+0000","last_undegraded":"2023-09-29T23:06:18.288529+0000","last_fullsized":"2023-09-29T23:06:18.288529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282000+0000","last_change":"2023-09-29T23:06:17.891835+0000","last_active":"2023-09-29T23:06:18.282000+0000","last_peered":"2023-09-29T23:06:18.282000+0000","last_clean":"2023-09-29T23:06:18.282000+0000","last_became_active":"2023-09-29T23:06:17.891563+0000","last_became_peered":"2023-09-29T23:06:17.891563+0000","last_unstale":"2023-09-29T23:06:18.282000+0000","last_undegraded":"2023-09-29T23:06:18.282000+0000","last_fullsized":"2023-09-29T23:06:18.282000+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:22.597416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292423+0000","last_change":"2023-09-29T23:06:18.289343+0000","last_active":"2023-09-29T23:06:18.292423+0000","last_peered":"2023-09-29T23:06:18.292423+0000","last_clean":"2023-09-29T23:06:18.292423+0000","last_became_active":"2023-09-29T23:06:18.289072+0000","last_became_peered":"2023-09-29T23:06:18.289072+0000","last_unstale":"2023-09-29T23:06:18.292423+0000","last_undegraded":"2023-09-29T23:06:18.292423+0000","last_fullsized":"2023-09-29T23:06:18.292423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:27:12.781947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282222+0000","last_change":"2023-09-29T23:06:17.891844+0000","last_active":"2023-09-29T23:06:18.282222+0000","last_peered":"2023-09-29T23:06:18.282222+0000","last_clean":"2023-09-29T23:06:18.282222+0000","last_became_active":"2023-09-29T23:06:17.891582+0000","last_became_peered":"2023-09-29T23:06:17.891582+0000","last_unstale":"2023-09-29T23:06:18.282222+0000","last_undegraded":"2023-09-29T23:06:18.282222+0000","last_fullsized":"2023-09-29T23:06:18.282222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:04:59.037943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287898+0000","last_change":"2023-09-29T23:06:18.285944+0000","last_active":"2023-09-29T23:06:18.287898+0000","last_peered":"2023-09-29T23:06:18.287898+0000","last_clean":"2023-09-29T23:06:18.287898+0000","last_became_active":"2023-09-29T23:06:18.285661+0000","last_became_peered":"2023-09-29T23:06:18.285661+0000","last_unstale":"2023-09-29T23:06:18.287898+0000","last_undegraded":"2023-09-29T23:06:18.287898+0000","last_fullsized":"2023-09-29T23:06:18.287898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:20:25.540032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.275819+0000","last_change":"2023-09-29T23:06:17.889165+0000","last_active":"2023-09-29T23:06:18.275819+0000","last_peered":"2023-09-29T23:06:18.275819+0000","last_clean":"2023-09-29T23:06:18.275819+0000","last_became_active":"2023-09-29T23:06:17.888842+0000","last_became_peered":"2023-09-29T23:06:17.888842+0000","last_unstale":"2023-09-29T23:06:18.275819+0000","last_undegraded":"2023-09-29T23:06:18.275819+0000","last_fullsized":"2023-09-29T23:06:18.275819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:41:35.202255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.287772+0000","last_change":"2023-09-29T23:06:18.277402+0000","last_active":"2023-09-29T23:06:18.287772+0000","last_peered":"2023-09-29T23:06:18.287772+0000","last_clean":"2023-09-29T23:06:18.287772+0000","last_became_active":"2023-09-29T23:06:18.277159+0000","last_became_peered":"2023-09-29T23:06:18.277159+0000","last_unstale":"2023-09-29T23:06:18.287772+0000","last_undegraded":"2023-09-29T23:06:18.287772+0000","last_fullsized":"2023-09-29T23:06:18.287772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:53:33.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273008+0000","last_change":"2023-09-29T23:06:17.890237+0000","last_active":"2023-09-29T23:06:18.273008+0000","last_peered":"2023-09-29T23:06:18.273008+0000","last_clean":"2023-09-29T23:06:18.273008+0000","last_became_active":"2023-09-29T23:06:17.889971+0000","last_became_peered":"2023-09-29T23:06:17.889971+0000","last_unstale":"2023-09-29T23:06:18.273008+0000","last_undegraded":"2023-09-29T23:06:18.273008+0000","last_fullsized":"2023-09-29T23:06:18.273008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:57.168709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.282466+0000","last_change":"2023-09-29T23:06:18.276262+0000","last_active":"2023-09-29T23:06:18.282466+0000","last_peered":"2023-09-29T23:06:18.282466+0000","last_clean":"2023-09-29T23:06:18.282466+0000","last_became_active":"2023-09-29T23:06:18.276040+0000","last_became_peered":"2023-09-29T23:06:18.276040+0000","last_unstale":"2023-09-29T23:06:18.282466+0000","last_undegraded":"2023-09-29T23:06:18.282466+0000","last_fullsized":"2023-09-29T23:06:18.282466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:08:18.940128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260885+0000","last_change":"2023-09-29T23:06:17.888828+0000","last_active":"2023-09-29T23:06:18.260885+0000","last_peered":"2023-09-29T23:06:18.260885+0000","last_clean":"2023-09-29T23:06:18.260885+0000","last_became_active":"2023-09-29T23:06:17.888525+0000","last_became_peered":"2023-09-29T23:06:17.888525+0000","last_unstale":"2023-09-29T23:06:18.260885+0000","last_undegraded":"2023-09-29T23:06:18.260885+0000","last_fullsized":"2023-09-29T23:06:18.260885+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:18:04.884961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285419+0000","last_change":"2023-09-29T23:06:18.283530+0000","last_active":"2023-09-29T23:06:18.285419+0000","last_peered":"2023-09-29T23:06:18.285419+0000","last_clean":"2023-09-29T23:06:18.285419+0000","last_became_active":"2023-09-29T23:06:18.281248+0000","last_became_peered":"2023-09-29T23:06:18.281248+0000","last_unstale":"2023-09-29T23:06:18.285419+0000","last_undegraded":"2023-09-29T23:06:18.285419+0000","last_fullsized":"2023-09-29T23:06:18.285419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:39:23.599215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.289625+0000","last_change":"2023-09-29T23:06:17.894026+0000","last_active":"2023-09-29T23:06:18.289625+0000","last_peered":"2023-09-29T23:06:18.289625+0000","last_clean":"2023-09-29T23:06:18.289625+0000","last_became_active":"2023-09-29T23:06:17.893757+0000","last_became_peered":"2023-09-29T23:06:17.893757+0000","last_unstale":"2023-09-29T23:06:18.289625+0000","last_undegraded":"2023-09-29T23:06:18.289625+0000","last_fullsized":"2023-09-29T23:06:18.289625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:31:14.359054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286633+0000","last_change":"2023-09-29T23:06:18.278135+0000","last_active":"2023-09-29T23:06:18.286633+0000","last_peered":"2023-09-29T23:06:18.286633+0000","last_clean":"2023-09-29T23:06:18.286633+0000","last_became_active":"2023-09-29T23:06:18.277865+0000","last_became_peered":"2023-09-29T23:06:18.277865+0000","last_unstale":"2023-09-29T23:06:18.286633+0000","last_undegraded":"2023-09-29T23:06:18.286633+0000","last_fullsized":"2023-09-29T23:06:18.286633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260719+0000","last_change":"2023-09-29T23:06:17.891680+0000","last_active":"2023-09-29T23:06:18.260719+0000","last_peered":"2023-09-29T23:06:18.260719+0000","last_clean":"2023-09-29T23:06:18.260719+0000","last_became_active":"2023-09-29T23:06:17.891413+0000","last_became_peered":"2023-09-29T23:06:17.891413+0000","last_unstale":"2023-09-29T23:06:18.260719+0000","last_undegraded":"2023-09-29T23:06:18.260719+0000","last_fullsized":"2023-09-29T23:06:18.260719+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:19.905553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.269364+0000","last_change":"2023-09-29T23:06:18.268288+0000","last_active":"2023-09-29T23:06:18.269364+0000","last_peered":"2023-09-29T23:06:18.269364+0000","last_clean":"2023-09-29T23:06:18.269364+0000","last_became_active":"2023-09-29T23:06:18.268079+0000","last_became_peered":"2023-09-29T23:06:18.268079+0000","last_unstale":"2023-09-29T23:06:18.269364+0000","last_undegraded":"2023-09-29T23:06:18.269364+0000","last_fullsized":"2023-09-29T23:06:18.269364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274260+0000","last_change":"2023-09-29T23:06:17.892193+0000","last_active":"2023-09-29T23:06:18.274260+0000","last_peered":"2023-09-29T23:06:18.274260+0000","last_clean":"2023-09-29T23:06:18.274260+0000","last_became_active":"2023-09-29T23:06:17.891898+0000","last_became_peered":"2023-09-29T23:06:17.891898+0000","last_unstale":"2023-09-29T23:06:18.274260+0000","last_undegraded":"2023-09-29T23:06:18.274260+0000","last_fullsized":"2023-09-29T23:06:18.274260+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:28:41.221257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.276612+0000","last_change":"2023-09-29T23:06:18.271513+0000","last_active":"2023-09-29T23:06:18.276612+0000","last_peered":"2023-09-29T23:06:18.276612+0000","last_clean":"2023-09-29T23:06:18.276612+0000","last_became_active":"2023-09-29T23:06:18.271257+0000","last_became_peered":"2023-09-29T23:06:18.271257+0000","last_unstale":"2023-09-29T23:06:18.276612+0000","last_undegraded":"2023-09-29T23:06:18.276612+0000","last_fullsized":"2023-09-29T23:06:18.276612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:23.301482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273504+0000","last_change":"2023-09-29T23:06:17.893616+0000","last_active":"2023-09-29T23:06:18.273504+0000","last_peered":"2023-09-29T23:06:18.273504+0000","last_clean":"2023-09-29T23:06:18.273504+0000","last_became_active":"2023-09-29T23:06:17.893285+0000","last_became_peered":"2023-09-29T23:06:17.893285+0000","last_unstale":"2023-09-29T23:06:18.273504+0000","last_undegraded":"2023-09-29T23:06:18.273504+0000","last_fullsized":"2023-09-29T23:06:18.273504+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283690+0000","last_change":"2023-09-29T23:06:18.277473+0000","last_active":"2023-09-29T23:06:18.283690+0000","last_peered":"2023-09-29T23:06:18.283690+0000","last_clean":"2023-09-29T23:06:18.283690+0000","last_became_active":"2023-09-29T23:06:18.276953+0000","last_became_peered":"2023-09-29T23:06:18.276953+0000","last_unstale":"2023-09-29T23:06:18.283690+0000","last_undegraded":"2023-09-29T23:06:18.283690+0000","last_fullsized":"2023-09-29T23:06:18.283690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:22.562666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264133+0000","last_change":"2023-09-29T23:06:13.639723+0000","last_active":"2023-09-29T23:06:18.264133+0000","last_peered":"2023-09-29T23:06:18.264133+0000","last_clean":"2023-09-29T23:06:18.264133+0000","last_became_active":"2023-09-29T23:06:13.639406+0000","last_became_peered":"2023-09-29T23:06:13.639406+0000","last_unstale":"2023-09-29T23:06:18.264133+0000","last_undegraded":"2023-09-29T23:06:18.264133+0000","last_fullsized":"2023-09-29T23:06:18.264133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:07:23.007465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261412+0000","last_change":"2023-09-29T23:06:17.894258+0000","last_active":"2023-09-29T23:06:18.261412+0000","last_peered":"2023-09-29T23:06:18.261412+0000","last_clean":"2023-09-29T23:06:18.261412+0000","last_became_active":"2023-09-29T23:06:17.893935+0000","last_became_peered":"2023-09-29T23:06:17.893935+0000","last_unstale":"2023-09-29T23:06:18.261412+0000","last_undegraded":"2023-09-29T23:06:18.261412+0000","last_fullsized":"2023-09-29T23:06:18.261412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:30:41.674282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280296+0000","last_change":"2023-09-29T23:06:18.271186+0000","last_active":"2023-09-29T23:06:18.280296+0000","last_peered":"2023-09-29T23:06:18.280296+0000","last_clean":"2023-09-29T23:06:18.280296+0000","last_became_active":"2023-09-29T23:06:18.270718+0000","last_became_peered":"2023-09-29T23:06:18.270718+0000","last_unstale":"2023-09-29T23:06:18.280296+0000","last_undegraded":"2023-09-29T23:06:18.280296+0000","last_fullsized":"2023-09-29T23:06:18.280296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:31:58.170509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260686+0000","last_change":"2023-09-29T23:06:13.380282+0000","last_active":"2023-09-29T23:06:18.260686+0000","last_peered":"2023-09-29T23:06:18.260686+0000","last_clean":"2023-09-29T23:06:18.260686+0000","last_became_active":"2023-09-29T23:06:13.379948+0000","last_became_peered":"2023-09-29T23:06:13.379948+0000","last_unstale":"2023-09-29T23:06:18.260686+0000","last_undegraded":"2023-09-29T23:06:18.260686+0000","last_fullsized":"2023-09-29T23:06:18.260686+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:02.759420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261279+0000","last_change":"2023-09-29T23:06:17.884199+0000","last_active":"2023-09-29T23:06:18.261279+0000","last_peered":"2023-09-29T23:06:18.261279+0000","last_clean":"2023-09-29T23:06:18.261279+0000","last_became_active":"2023-09-29T23:06:17.883932+0000","last_became_peered":"2023-09-29T23:06:17.883932+0000","last_unstale":"2023-09-29T23:06:18.261279+0000","last_undegraded":"2023-09-29T23:06:18.261279+0000","last_fullsized":"2023-09-29T23:06:18.261279+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:43:39.182159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283703+0000","last_change":"2023-09-29T23:06:18.277498+0000","last_active":"2023-09-29T23:06:18.283703+0000","last_peered":"2023-09-29T23:06:18.283703+0000","last_clean":"2023-09-29T23:06:18.283703+0000","last_became_active":"2023-09-29T23:06:18.276885+0000","last_became_peered":"2023-09-29T23:06:18.276885+0000","last_unstale":"2023-09-29T23:06:18.283703+0000","last_undegraded":"2023-09-29T23:06:18.283703+0000","last_fullsized":"2023-09-29T23:06:18.283703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.259675+0000","last_change":"2023-09-29T23:06:13.639753+0000","last_active":"2023-09-29T23:06:18.259675+0000","last_peered":"2023-09-29T23:06:18.259675+0000","last_clean":"2023-09-29T23:06:18.259675+0000","last_became_active":"2023-09-29T23:06:13.639436+0000","last_became_peered":"2023-09-29T23:06:13.639436+0000","last_unstale":"2023-09-29T23:06:18.259675+0000","last_undegraded":"2023-09-29T23:06:18.259675+0000","last_fullsized":"2023-09-29T23:06:18.259675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:09:57.060849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261100+0000","last_change":"2023-09-29T23:06:17.885616+0000","last_active":"2023-09-29T23:06:18.261100+0000","last_peered":"2023-09-29T23:06:18.261100+0000","last_clean":"2023-09-29T23:06:18.261100+0000","last_became_active":"2023-09-29T23:06:17.885342+0000","last_became_peered":"2023-09-29T23:06:17.885342+0000","last_unstale":"2023-09-29T23:06:18.261100+0000","last_undegraded":"2023-09-29T23:06:18.261100+0000","last_fullsized":"2023-09-29T23:06:18.261100+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:22:05.264555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280779+0000","last_change":"2023-09-29T23:06:18.277223+0000","last_active":"2023-09-29T23:06:18.280779+0000","last_peered":"2023-09-29T23:06:18.280779+0000","last_clean":"2023-09-29T23:06:18.280779+0000","last_became_active":"2023-09-29T23:06:18.276548+0000","last_became_peered":"2023-09-29T23:06:18.276548+0000","last_unstale":"2023-09-29T23:06:18.280779+0000","last_undegraded":"2023-09-29T23:06:18.280779+0000","last_fullsized":"2023-09-29T23:06:18.280779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261249+0000","last_change":"2023-09-29T23:06:13.380967+0000","last_active":"2023-09-29T23:06:18.261249+0000","last_peered":"2023-09-29T23:06:18.261249+0000","last_clean":"2023-09-29T23:06:18.261249+0000","last_became_active":"2023-09-29T23:06:13.379890+0000","last_became_peered":"2023-09-29T23:06:13.379890+0000","last_unstale":"2023-09-29T23:06:18.261249+0000","last_undegraded":"2023-09-29T23:06:18.261249+0000","last_fullsized":"2023-09-29T23:06:18.261249+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:33:46.738824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260986+0000","last_change":"2023-09-29T23:06:09.717533+0000","last_active":"2023-09-29T23:06:18.260986+0000","last_peered":"2023-09-29T23:06:18.260986+0000","last_clean":"2023-09-29T23:06:18.260986+0000","last_became_active":"2023-09-29T23:06:09.716536+0000","last_became_peered":"2023-09-29T23:06:09.716536+0000","last_unstale":"2023-09-29T23:06:18.260986+0000","last_undegraded":"2023-09-29T23:06:18.260986+0000","last_fullsized":"2023-09-29T23:06:18.260986+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:08.290533+0000","last_clean_scrub_stamp":"2023-09-29T23:06:08.290533+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:44:31.128783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274816+0000","last_change":"2023-09-29T23:06:17.892420+0000","last_active":"2023-09-29T23:06:18.274816+0000","last_peered":"2023-09-29T23:06:18.274816+0000","last_clean":"2023-09-29T23:06:18.274816+0000","last_became_active":"2023-09-29T23:06:17.892121+0000","last_became_peered":"2023-09-29T23:06:17.892121+0000","last_unstale":"2023-09-29T23:06:18.274816+0000","last_undegraded":"2023-09-29T23:06:18.274816+0000","last_fullsized":"2023-09-29T23:06:18.274816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:00:05.872821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283140+0000","last_change":"2023-09-29T23:06:18.281364+0000","last_active":"2023-09-29T23:06:18.283140+0000","last_peered":"2023-09-29T23:06:18.283140+0000","last_clean":"2023-09-29T23:06:18.283140+0000","last_became_active":"2023-09-29T23:06:18.280821+0000","last_became_peered":"2023-09-29T23:06:18.280821+0000","last_unstale":"2023-09-29T23:06:18.283140+0000","last_undegraded":"2023-09-29T23:06:18.283140+0000","last_fullsized":"2023-09-29T23:06:18.283140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:53:26.396498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260488+0000","last_change":"2023-09-29T23:06:13.380209+0000","last_active":"2023-09-29T23:06:18.260488+0000","last_peered":"2023-09-29T23:06:18.260488+0000","last_clean":"2023-09-29T23:06:18.260488+0000","last_became_active":"2023-09-29T23:06:13.379845+0000","last_became_peered":"2023-09-29T23:06:13.379845+0000","last_unstale":"2023-09-29T23:06:18.260488+0000","last_undegraded":"2023-09-29T23:06:18.260488+0000","last_fullsized":"2023-09-29T23:06:18.260488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261391+0000","last_change":"2023-09-29T23:06:17.891971+0000","last_active":"2023-09-29T23:06:18.261391+0000","last_peered":"2023-09-29T23:06:18.261391+0000","last_clean":"2023-09-29T23:06:18.261391+0000","last_became_active":"2023-09-29T23:06:17.891695+0000","last_became_peered":"2023-09-29T23:06:17.891695+0000","last_unstale":"2023-09-29T23:06:18.261391+0000","last_undegraded":"2023-09-29T23:06:18.261391+0000","last_fullsized":"2023-09-29T23:06:18.261391+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:19.939194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292294+0000","last_change":"2023-09-29T23:06:18.288649+0000","last_active":"2023-09-29T23:06:18.292294+0000","last_peered":"2023-09-29T23:06:18.292294+0000","last_clean":"2023-09-29T23:06:18.292294+0000","last_became_active":"2023-09-29T23:06:18.288351+0000","last_became_peered":"2023-09-29T23:06:18.288351+0000","last_unstale":"2023-09-29T23:06:18.292294+0000","last_undegraded":"2023-09-29T23:06:18.292294+0000","last_fullsized":"2023-09-29T23:06:18.292294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:32:20.060786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.261483+0000","last_change":"2023-09-29T23:06:13.380613+0000","last_active":"2023-09-29T23:06:18.261483+0000","last_peered":"2023-09-29T23:06:18.261483+0000","last_clean":"2023-09-29T23:06:18.261483+0000","last_became_active":"2023-09-29T23:06:13.380347+0000","last_became_peered":"2023-09-29T23:06:13.380347+0000","last_unstale":"2023-09-29T23:06:18.261483+0000","last_undegraded":"2023-09-29T23:06:18.261483+0000","last_fullsized":"2023-09-29T23:06:18.261483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:35.568211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.264019+0000","last_change":"2023-09-29T23:06:17.885489+0000","last_active":"2023-09-29T23:06:18.264019+0000","last_peered":"2023-09-29T23:06:18.264019+0000","last_clean":"2023-09-29T23:06:18.264019+0000","last_became_active":"2023-09-29T23:06:17.884966+0000","last_became_peered":"2023-09-29T23:06:17.884966+0000","last_unstale":"2023-09-29T23:06:18.264019+0000","last_undegraded":"2023-09-29T23:06:18.264019+0000","last_fullsized":"2023-09-29T23:06:18.264019+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286897+0000","last_change":"2023-09-29T23:06:18.282836+0000","last_active":"2023-09-29T23:06:18.286897+0000","last_peered":"2023-09-29T23:06:18.286897+0000","last_clean":"2023-09-29T23:06:18.286897+0000","last_became_active":"2023-09-29T23:06:18.282215+0000","last_became_peered":"2023-09-29T23:06:18.282215+0000","last_unstale":"2023-09-29T23:06:18.286897+0000","last_undegraded":"2023-09-29T23:06:18.286897+0000","last_fullsized":"2023-09-29T23:06:18.286897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:54:51.392629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288992+0000","last_change":"2023-09-29T23:06:17.894025+0000","last_active":"2023-09-29T23:06:18.288992+0000","last_peered":"2023-09-29T23:06:18.288992+0000","last_clean":"2023-09-29T23:06:18.288992+0000","last_became_active":"2023-09-29T23:06:17.893701+0000","last_became_peered":"2023-09-29T23:06:17.893701+0000","last_unstale":"2023-09-29T23:06:18.288992+0000","last_undegraded":"2023-09-29T23:06:18.288992+0000","last_fullsized":"2023-09-29T23:06:18.288992+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:57.077867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.270574+0000","last_change":"2023-09-29T23:06:13.379718+0000","last_active":"2023-09-29T23:06:18.270574+0000","last_peered":"2023-09-29T23:06:18.270574+0000","last_clean":"2023-09-29T23:06:18.270574+0000","last_became_active":"2023-09-29T23:06:13.379450+0000","last_became_peered":"2023-09-29T23:06:13.379450+0000","last_unstale":"2023-09-29T23:06:18.270574+0000","last_undegraded":"2023-09-29T23:06:18.270574+0000","last_fullsized":"2023-09-29T23:06:18.270574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:53:14.849675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286470+0000","last_change":"2023-09-29T23:06:18.283988+0000","last_active":"2023-09-29T23:06:18.286470+0000","last_peered":"2023-09-29T23:06:18.286470+0000","last_clean":"2023-09-29T23:06:18.286470+0000","last_became_active":"2023-09-29T23:06:18.283448+0000","last_became_peered":"2023-09-29T23:06:18.283448+0000","last_unstale":"2023-09-29T23:06:18.286470+0000","last_undegraded":"2023-09-29T23:06:18.286470+0000","last_fullsized":"2023-09-29T23:06:18.286470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:00:15.860186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.260973+0000","last_change":"2023-09-29T23:06:13.382829+0000","last_active":"2023-09-29T23:06:18.260973+0000","last_peered":"2023-09-29T23:06:18.260973+0000","last_clean":"2023-09-29T23:06:18.260973+0000","last_became_active":"2023-09-29T23:06:13.382590+0000","last_became_peered":"2023-09-29T23:06:13.382590+0000","last_unstale":"2023-09-29T23:06:18.260973+0000","last_undegraded":"2023-09-29T23:06:18.260973+0000","last_fullsized":"2023-09-29T23:06:18.260973+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:12.357531+0000","last_clean_scrub_stamp":"2023-09-29T23:06:12.357531+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:06.822425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-09-29T23:06:19.267536+0000","last_change":"2023-09-29T23:06:17.881636+0000","last_active":"2023-09-29T23:06:19.267536+0000","last_peered":"2023-09-29T23:06:19.267536+0000","last_clean":"2023-09-29T23:06:19.267536+0000","last_became_active":"2023-09-29T23:06:17.881343+0000","last_became_peered":"2023-09-29T23:06:17.881343+0000","last_unstale":"2023-09-29T23:06:19.267536+0000","last_undegraded":"2023-09-29T23:06:19.267536+0000","last_fullsized":"2023-09-29T23:06:19.267536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:55:34.442607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.274553+0000","last_change":"2023-09-29T23:06:18.268514+0000","last_active":"2023-09-29T23:06:18.274553+0000","last_peered":"2023-09-29T23:06:18.274553+0000","last_clean":"2023-09-29T23:06:18.274553+0000","last_became_active":"2023-09-29T23:06:18.268089+0000","last_became_peered":"2023-09-29T23:06:18.268089+0000","last_unstale":"2023-09-29T23:06:18.274553+0000","last_undegraded":"2023-09-29T23:06:18.274553+0000","last_fullsized":"2023-09-29T23:06:18.274553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:22.930009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288913+0000","last_change":"2023-09-29T23:06:17.892888+0000","last_active":"2023-09-29T23:06:18.288913+0000","last_peered":"2023-09-29T23:06:18.288913+0000","last_clean":"2023-09-29T23:06:18.288913+0000","last_became_active":"2023-09-29T23:06:17.892666+0000","last_became_peered":"2023-09-29T23:06:17.892666+0000","last_unstale":"2023-09-29T23:06:18.288913+0000","last_undegraded":"2023-09-29T23:06:18.288913+0000","last_fullsized":"2023-09-29T23:06:18.288913+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:47:21.928210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285341+0000","last_change":"2023-09-29T23:06:18.282028+0000","last_active":"2023-09-29T23:06:18.285341+0000","last_peered":"2023-09-29T23:06:18.285341+0000","last_clean":"2023-09-29T23:06:18.285341+0000","last_became_active":"2023-09-29T23:06:18.281546+0000","last_became_peered":"2023-09-29T23:06:18.281546+0000","last_unstale":"2023-09-29T23:06:18.285341+0000","last_undegraded":"2023-09-29T23:06:18.285341+0000","last_fullsized":"2023-09-29T23:06:18.285341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:20.907352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285659+0000","last_change":"2023-09-29T23:06:17.891510+0000","last_active":"2023-09-29T23:06:18.285659+0000","last_peered":"2023-09-29T23:06:18.285659+0000","last_clean":"2023-09-29T23:06:18.285659+0000","last_became_active":"2023-09-29T23:06:17.891217+0000","last_became_peered":"2023-09-29T23:06:17.891217+0000","last_unstale":"2023-09-29T23:06:18.285659+0000","last_undegraded":"2023-09-29T23:06:18.285659+0000","last_fullsized":"2023-09-29T23:06:18.285659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:45:22.299682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.285386+0000","last_change":"2023-09-29T23:06:18.283220+0000","last_active":"2023-09-29T23:06:18.285386+0000","last_peered":"2023-09-29T23:06:18.285386+0000","last_clean":"2023-09-29T23:06:18.285386+0000","last_became_active":"2023-09-29T23:06:18.281619+0000","last_became_peered":"2023-09-29T23:06:18.281619+0000","last_unstale":"2023-09-29T23:06:18.285386+0000","last_undegraded":"2023-09-29T23:06:18.285386+0000","last_fullsized":"2023-09-29T23:06:18.285386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:05:00.039742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.272636+0000","last_change":"2023-09-29T23:06:17.887773+0000","last_active":"2023-09-29T23:06:18.272636+0000","last_peered":"2023-09-29T23:06:18.272636+0000","last_clean":"2023-09-29T23:06:18.272636+0000","last_became_active":"2023-09-29T23:06:17.887578+0000","last_became_peered":"2023-09-29T23:06:17.887578+0000","last_unstale":"2023-09-29T23:06:18.272636+0000","last_undegraded":"2023-09-29T23:06:18.272636+0000","last_fullsized":"2023-09-29T23:06:18.272636+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:04:22.787417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288175+0000","last_change":"2023-09-29T23:06:18.287213+0000","last_active":"2023-09-29T23:06:18.288175+0000","last_peered":"2023-09-29T23:06:18.288175+0000","last_clean":"2023-09-29T23:06:18.288175+0000","last_became_active":"2023-09-29T23:06:18.286861+0000","last_became_peered":"2023-09-29T23:06:18.286861+0000","last_unstale":"2023-09-29T23:06:18.288175+0000","last_undegraded":"2023-09-29T23:06:18.288175+0000","last_fullsized":"2023-09-29T23:06:18.288175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:44:31.777462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.286577+0000","last_change":"2023-09-29T23:06:17.893529+0000","last_active":"2023-09-29T23:06:18.286577+0000","last_peered":"2023-09-29T23:06:18.286577+0000","last_clean":"2023-09-29T23:06:18.286577+0000","last_became_active":"2023-09-29T23:06:17.893269+0000","last_became_peered":"2023-09-29T23:06:17.893269+0000","last_unstale":"2023-09-29T23:06:18.286577+0000","last_undegraded":"2023-09-29T23:06:18.286577+0000","last_fullsized":"2023-09-29T23:06:18.286577+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288800+0000","last_change":"2023-09-29T23:06:18.286214+0000","last_active":"2023-09-29T23:06:18.288800+0000","last_peered":"2023-09-29T23:06:18.288800+0000","last_clean":"2023-09-29T23:06:18.288800+0000","last_became_active":"2023-09-29T23:06:18.285773+0000","last_became_peered":"2023-09-29T23:06:18.285773+0000","last_unstale":"2023-09-29T23:06:18.288800+0000","last_undegraded":"2023-09-29T23:06:18.288800+0000","last_fullsized":"2023-09-29T23:06:18.288800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:46:43.840774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.278612+0000","last_change":"2023-09-29T23:06:17.892342+0000","last_active":"2023-09-29T23:06:18.278612+0000","last_peered":"2023-09-29T23:06:18.278612+0000","last_clean":"2023-09-29T23:06:18.278612+0000","last_became_active":"2023-09-29T23:06:17.891454+0000","last_became_peered":"2023-09-29T23:06:17.891454+0000","last_unstale":"2023-09-29T23:06:18.278612+0000","last_undegraded":"2023-09-29T23:06:18.278612+0000","last_fullsized":"2023-09-29T23:06:18.278612+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:16.860955+0000","last_clean_scrub_stamp":"2023-09-29T23:06:16.860955+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:26.076042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283064+0000","last_change":"2023-09-29T23:06:18.280840+0000","last_active":"2023-09-29T23:06:18.283064+0000","last_peered":"2023-09-29T23:06:18.283064+0000","last_clean":"2023-09-29T23:06:18.283064+0000","last_became_active":"2023-09-29T23:06:18.280656+0000","last_became_peered":"2023-09-29T23:06:18.280656+0000","last_unstale":"2023-09-29T23:06:18.283064+0000","last_undegraded":"2023-09-29T23:06:18.283064+0000","last_fullsized":"2023-09-29T23:06:18.283064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:13.688293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.273870+0000","last_change":"2023-09-29T23:06:18.272838+0000","last_active":"2023-09-29T23:06:18.273870+0000","last_peered":"2023-09-29T23:06:18.273870+0000","last_clean":"2023-09-29T23:06:18.273870+0000","last_became_active":"2023-09-29T23:06:18.272444+0000","last_became_peered":"2023-09-29T23:06:18.272444+0000","last_unstale":"2023-09-29T23:06:18.273870+0000","last_undegraded":"2023-09-29T23:06:18.273870+0000","last_fullsized":"2023-09-29T23:06:18.273870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:25:46.885392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.288812+0000","last_change":"2023-09-29T23:06:18.286345+0000","last_active":"2023-09-29T23:06:18.288812+0000","last_peered":"2023-09-29T23:06:18.288812+0000","last_clean":"2023-09-29T23:06:18.288812+0000","last_became_active":"2023-09-29T23:06:18.285985+0000","last_became_peered":"2023-09-29T23:06:18.285985+0000","last_unstale":"2023-09-29T23:06:18.288812+0000","last_undegraded":"2023-09-29T23:06:18.288812+0000","last_fullsized":"2023-09-29T23:06:18.288812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:35.802092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.280311+0000","last_change":"2023-09-29T23:06:18.271411+0000","last_active":"2023-09-29T23:06:18.280311+0000","last_peered":"2023-09-29T23:06:18.280311+0000","last_clean":"2023-09-29T23:06:18.280311+0000","last_became_active":"2023-09-29T23:06:18.270318+0000","last_became_peered":"2023-09-29T23:06:18.270318+0000","last_unstale":"2023-09-29T23:06:18.280311+0000","last_undegraded":"2023-09-29T23:06:18.280311+0000","last_fullsized":"2023-09-29T23:06:18.280311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:56.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.292425+0000","last_change":"2023-09-29T23:06:18.289646+0000","last_active":"2023-09-29T23:06:18.292425+0000","last_peered":"2023-09-29T23:06:18.292425+0000","last_clean":"2023-09-29T23:06:18.292425+0000","last_became_active":"2023-09-29T23:06:18.289304+0000","last_became_peered":"2023-09-29T23:06:18.289304+0000","last_unstale":"2023-09-29T23:06:18.292425+0000","last_undegraded":"2023-09-29T23:06:18.292425+0000","last_fullsized":"2023-09-29T23:06:18.292425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:18:01.223023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-29T23:06:18.283682+0000","last_change":"2023-09-29T23:06:18.277502+0000","last_active":"2023-09-29T23:06:18.283682+0000","last_peered":"2023-09-29T23:06:18.283682+0000","last_clean":"2023-09-29T23:06:18.283682+0000","last_became_active":"2023-09-29T23:06:18.277224+0000","last_became_peered":"2023-09-29T23:06:18.277224+0000","last_unstale":"2023-09-29T23:06:18.283682+0000","last_undegraded":"2023-09-29T23:06:18.283682+0000","last_fullsized":"2023-09-29T23:06:18.283682+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T23:06:17.862754+0000","last_clean_scrub_stamp":"2023-09-29T23:06:17.862754+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:48:40.869347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":10802,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":188416,"data_stored":60567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8960,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362880,"statfs":{"total":96636764160,"available":96627589120,"internally_reserved":0,"allocated":253952,"data_stored":652921,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8588,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363252,"statfs":{"total":96636764160,"available":96627970048,"internally_reserved":0,"allocated":200704,"data_stored":62035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9076,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362764,"statfs":{"total":96636764160,"available":96627470336,"internally_reserved":0,"allocated":241664,"data_stored":651317,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5401,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T23:06:31.063 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-29T23:06:31.063 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-29T23:06:31.063 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-29T23:06:31.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:31.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.214+0000 7f996fe87700 1 Processor -- start 2023-09-29T23:06:31.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.214+0000 7f996fe87700 1 -- start start 2023-09-29T23:06:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996fe87700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99680630e0 0x7f996809c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996fe87700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996809e600 con 0x7f996809f580 2023-09-29T23:06:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996fe87700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680a2d20 con 0x7f99680634b0 2023-09-29T23:06:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996fe87700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996809d8d0 con 0x7f99680630e0 2023-09-29T23:06:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996dc23700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99680630e0 0x7f996809c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996e424700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f996809f580 0x7f996809ccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44536/0 (socket says 172.21.15.121:44536) 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996e424700 1 -- 172.21.15.121:0/990505553 learned_addr learned my addr 172.21.15.121:0/990505553 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996dc23700 1 -- 172.21.15.121:0/990505553 >> v1:172.21.15.123:6789/0 conn(0x7f99680634b0 legacy=0x7f996809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996cc21700 1 -- 172.21.15.121:0/990505553 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037271081 0 0) 0x7f996809e600 con 0x7f996809f580 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996dc23700 1 -- 172.21.15.121:0/990505553 >> v1:172.21.15.121:6789/0 conn(0x7f996809f580 legacy=0x7f996809ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996dc23700 1 -- 172.21.15.121:0/990505553 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996809db70 con 0x7f99680630e0 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.215+0000 7f996dc23700 1 --2- 172.21.15.121:0/990505553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99680630e0 0x7f996809c070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9958009900 tx=0x7f9958009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e287eee80a8f8190 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996cc21700 1 -- 172.21.15.121:0/990505553 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958009ea0 con 0x7f99680630e0 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996cc21700 1 -- 172.21.15.121:0/990505553 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9958005c10 con 0x7f99680630e0 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996cc21700 1 -- 172.21.15.121:0/990505553 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800b610 con 0x7f99680630e0 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996fe87700 1 -- 172.21.15.121:0/990505553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99680630e0 msgr2=0x7f996809c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996fe87700 1 --2- 172.21.15.121:0/990505553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99680630e0 0x7f996809c070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9958009900 tx=0x7f9958009120 comp rx=0 tx=0).stop 2023-09-29T23:06:31.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996fe87700 1 -- 172.21.15.121:0/990505553 shutdown_connections 2023-09-29T23:06:31.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996fe87700 1 --2- 172.21.15.121:0/990505553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99680630e0 0x7f996809c070 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996fe87700 1 -- 172.21.15.121:0/990505553 >> 172.21.15.121:0/990505553 conn(0x7f9968069e90 msgr2=0x7f99680b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:31.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996fe87700 1 -- 172.21.15.121:0/990505553 shutdown_connections 2023-09-29T23:06:31.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.216+0000 7f996fe87700 1 -- 172.21.15.121:0/990505553 wait complete. 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.217+0000 7f996fe87700 1 Processor -- start 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.217+0000 7f996fe87700 1 -- start start 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996fe87700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99680630e0 0x7f99680ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996fe87700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680634b0 0x7f9968140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996fe87700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f996809f580 0x7f9968153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996fe87700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99680aea80 con 0x7f99680630e0 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996fe87700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99681411d0 con 0x7f99680634b0 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996fe87700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99680aef00 con 0x7f996809f580 2023-09-29T23:06:31.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996dc23700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99680630e0 0x7f99680ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996e424700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f996809f580 0x7f9968153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996dc23700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99680630e0 0x7f99680ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58708/0 (socket says 172.21.15.121:58708) 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996dc23700 1 -- 172.21.15.121:0/2249893901 learned_addr learned my addr 172.21.15.121:0/2249893901 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996d422700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680634b0 0x7f9968140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996e424700 1 -- 172.21.15.121:0/2249893901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680634b0 msgr2=0x7f9968140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996e424700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680634b0 0x7f9968140c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996e424700 1 -- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99680630e0 msgr2=0x7f99680ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996e424700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99680630e0 0x7f99680ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996e424700 1 -- 172.21.15.121:0/2249893901 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580093c0 con 0x7f996809f580 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996dc23700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99680630e0 0x7f99680ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996e424700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f996809f580 0x7f9968153be0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f996000cb60 tx=0x7f99600018e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f995dffb700 1 -- 172.21.15.121:0/2249893901 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996000a7e0 con 0x7f996809f580 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f995dffb700 1 -- 172.21.15.121:0/2249893901 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996000a960 con 0x7f996809f580 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99681419b0 con 0x7f996809f580 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.218+0000 7f995dffb700 1 -- 172.21.15.121:0/2249893901 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996000ad20 con 0x7f996809f580 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.219+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99681542e0 con 0x7f996809f580 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.219+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996806ab60 con 0x7f996809f580 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.220+0000 7f995dffb700 1 -- 172.21.15.121:0/2249893901 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9960007300 con 0x7f996809f580 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.220+0000 7f995dffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.221+0000 7f995dffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.221+0000 7f995dffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.221+0000 7f995dffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:31.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.221+0000 7f995dffb700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f994c098780 0x7f994c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:31.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.221+0000 7f996dc23700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f994c098780 0x7f994c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:31.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.221+0000 7f995dffb700 1 -- 172.21.15.121:0/2249893901 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f99600c0620 con 0x7f996809f580 2023-09-29T23:06:31.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.221+0000 7f996dc23700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f994c098780 0x7f994c09ac30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9958003080 tx=0x7f9958015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:31.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.225+0000 7f995dffb700 1 -- 172.21.15.121:0/2249893901 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f996008dcd0 con 0x7f996809f580 2023-09-29T23:06:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.443+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f99681538b0 con 0x7f996809f580 2023-09-29T23:06:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.443+0000 7f995dffb700 1 -- 172.21.15.121:0/2249893901 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f99600912f0 con 0x7f996809f580 2023-09-29T23:06:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:31.443 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 10 mgrc shutdown 2023-09-29T23:06:31.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f994c098780 msgr2=0x7f994c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f994c098780 0x7f994c09ac30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9958003080 tx=0x7f9958015000 comp rx=0 tx=0).stop 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f996809f580 msgr2=0x7f9968153be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f996809f580 0x7f9968153be0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f996000cb60 tx=0x7f99600018e0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 shutdown_connections 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f994c098780 0x7f994c09ac30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f996809f580 0x7f9968153be0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99680630e0 0x7f99680ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 --2- 172.21.15.121:0/2249893901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680634b0 0x7f9968140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 >> 172.21.15.121:0/2249893901 conn(0x7f9968069e90 msgr2=0x7f99681420e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 shutdown_connections 2023-09-29T23:06:31.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:31.445+0000 7f996fe87700 1 -- 172.21.15.121:0/2249893901 wait complete. 2023-09-29T23:06:34.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:34.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.587+0000 7fd4d20a6700 1 Processor -- start 2023-09-29T23:06:34.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.587+0000 7fd4d20a6700 1 -- start start 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4d20a6700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd4cc06b470 0x7fd4cc06b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4d20a6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc13e6d0 0x7fd4cc13eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4d20a6700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4cc0b53f0 con 0x7fd4cc0b5020 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4d20a6700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4cc13e2e0 con 0x7fd4cc06b470 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4d20a6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc09e970 con 0x7fd4cc13e6d0 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc13e6d0 0x7fd4cc13eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc13e6d0 0x7fd4cc13eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36786/0 (socket says 172.21.15.121:36786) 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 -- 172.21.15.121:0/2790901105 learned_addr learned my addr 172.21.15.121:0/2790901105 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4caffd700 1 --2- 172.21.15.121:0/2790901105 >> v2:172.21.15.123:3300/0 conn(0x7fd4cc06b470 0x7fd4cc06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 -- 172.21.15.121:0/2790901105 >> v2:172.21.15.123:3300/0 conn(0x7fd4cc06b470 msgr2=0x7fd4cc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 --2- 172.21.15.121:0/2790901105 >> v2:172.21.15.123:3300/0 conn(0x7fd4cc06b470 0x7fd4cc06b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 -- 172.21.15.121:0/2790901105 >> v1:172.21.15.123:6789/0 conn(0x7fd4cc0b5020 legacy=0x7fd4cc09e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 -- 172.21.15.121:0/2790901105 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc06ae80 con 0x7fd4cc13e6d0 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.588+0000 7fd4cbfff700 1 --2- 172.21.15.121:0/2790901105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc13e6d0 0x7fd4cc13eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4009f20 tx=0x7fd4c4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2c2a7ca882488b6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4ca7fc700 1 -- 172.21.15.121:0/2790901105 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c400e070 con 0x7fd4cc13e6d0 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4ca7fc700 1 -- 172.21.15.121:0/2790901105 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c400ba40 con 0x7fd4cc13e6d0 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4ca7fc700 1 -- 172.21.15.121:0/2790901105 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c40092e0 con 0x7fd4cc13e6d0 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2790901105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc13e6d0 msgr2=0x7fd4cc13eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2790901105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc13e6d0 0x7fd4cc13eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4009f20 tx=0x7fd4c4009840 comp rx=0 tx=0).stop 2023-09-29T23:06:34.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2790901105 shutdown_connections 2023-09-29T23:06:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2790901105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc13e6d0 0x7fd4cc13eb30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2790901105 >> v2:172.21.15.123:3300/0 conn(0x7fd4cc06b470 0x7fd4cc06b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2790901105 >> 172.21.15.121:0/2790901105 conn(0x7fd4cc06d0b0 msgr2=0x7fd4cc06d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2790901105 shutdown_connections 2023-09-29T23:06:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.589+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2790901105 wait complete. 2023-09-29T23:06:34.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.590+0000 7fd4d20a6700 1 Processor -- start 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.590+0000 7fd4d20a6700 1 -- start start 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4d20a6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc06b470 0x7fd4cc13ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4d20a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4cc0b5020 0x7fd4cc0a3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4d20a6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc13e6d0 0x7fd4cc096620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4d20a6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc13b810 con 0x7fd4cc0b5020 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4d20a6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc095d00 con 0x7fd4cc13e6d0 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4d20a6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc13ba40 con 0x7fd4cc06b470 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4cb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc06b470 0x7fd4cc13ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4cc0b5020 0x7fd4cc0a3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:34.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4cb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc06b470 0x7fd4cc13ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36800/0 (socket says 172.21.15.121:36800) 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4cb7fe700 1 -- 172.21.15.121:0/2318261750 learned_addr learned my addr 172.21.15.121:0/2318261750 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4cbfff700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc13e6d0 0x7fd4cc096620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4caffd700 1 -- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc06b470 msgr2=0x7fd4cc13ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4caffd700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc06b470 0x7fd4cc13ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4caffd700 1 -- 172.21.15.121:0/2318261750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc13e6d0 msgr2=0x7fd4cc096620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4caffd700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc13e6d0 0x7fd4cc096620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4caffd700 1 -- 172.21.15.121:0/2318261750 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c4009ae0 con 0x7fd4cc0b5020 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4cb7fe700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc06b470 0x7fd4cc13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4caffd700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4cc0b5020 0x7fd4cc0a3350 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00c880 tx=0x7fd4bc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.591+0000 7fd4c37fe700 1 -- 172.21.15.121:0/2318261750 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc011600 con 0x7fd4cc0b5020 2023-09-29T23:06:34.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.592+0000 7fd4c37fe700 1 -- 172.21.15.121:0/2318261750 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc011760 con 0x7fd4cc0b5020 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.592+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc0a3890 con 0x7fd4cc0b5020 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.592+0000 7fd4c37fe700 1 -- 172.21.15.121:0/2318261750 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc011b20 con 0x7fd4cc0b5020 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.592+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc0a3fc0 con 0x7fd4cc0b5020 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.593+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4cc096490 con 0x7fd4cc0b5020 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.593+0000 7fd4c37fe700 1 -- 172.21.15.121:0/2318261750 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd4bc018350 con 0x7fd4cc0b5020 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.593+0000 7fd4c37fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.594+0000 7fd4c37fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.594+0000 7fd4c37fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.594+0000 7fd4c37fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.594+0000 7fd4c37fe700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd4ac098780 0x7fd4ac09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.594+0000 7fd4cb7fe700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd4ac098780 0x7fd4ac09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:34.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.594+0000 7fd4c37fe700 1 -- 172.21.15.121:0/2318261750 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fd4bc0c0370 con 0x7fd4cc0b5020 2023-09-29T23:06:34.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.595+0000 7fd4cb7fe700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd4ac098780 0x7fd4ac09ac30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009f60 tx=0x7fd4b40091a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:34.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.598+0000 7fd4c37fe700 1 -- 172.21.15.121:0/2318261750 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd4bc08d990 con 0x7fd4cc0b5020 2023-09-29T23:06:34.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.816+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd4cc071660 con 0x7fd4cc0b5020 2023-09-29T23:06:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.817+0000 7fd4c37fe700 1 -- 172.21.15.121:0/2318261750 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd4bc01e100 con 0x7fd4cc0b5020 2023-09-29T23:06:34.817 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:34.817 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.818+0000 7fd4d20a6700 10 mgrc shutdown 2023-09-29T23:06:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd4ac098780 msgr2=0x7fd4ac09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd4ac098780 0x7fd4ac09ac30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009f60 tx=0x7fd4b40091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4cc0b5020 msgr2=0x7fd4cc0a3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4cc0b5020 0x7fd4cc0a3350 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00c880 tx=0x7fd4bc008040 comp rx=0 tx=0).stop 2023-09-29T23:06:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 shutdown_connections 2023-09-29T23:06:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd4ac098780 0x7fd4ac09ac30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc13e6d0 0x7fd4cc096620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4cc06b470 0x7fd4cc13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 --2- 172.21.15.121:0/2318261750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4cc0b5020 0x7fd4cc0a3350 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 >> 172.21.15.121:0/2318261750 conn(0x7fd4cc06d0b0 msgr2=0x7fd4cc067b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 shutdown_connections 2023-09-29T23:06:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:34.819+0000 7fd4d20a6700 1 -- 172.21.15.121:0/2318261750 wait complete. 2023-09-29T23:06:37.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:37.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.961+0000 7eff886c2700 1 Processor -- start 2023-09-29T23:06:37.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.961+0000 7eff886c2700 1 -- start start 2023-09-29T23:06:37.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.961+0000 7eff886c2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7eff80141440 0x7eff800a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.961+0000 7eff886c2700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7eff800b3cf0 0x7eff8014b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.961+0000 7eff886c2700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff80141810 con 0x7eff8009b770 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.961+0000 7eff886c2700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff8009a7f0 con 0x7eff800b3cf0 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.961+0000 7eff886c2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff800a4df0 con 0x7eff80141440 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7eff800b3cf0 0x7eff8014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7eff800b3cf0 0x7eff8014b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58152/0 (socket says 172.21.15.121:58152) 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 -- 172.21.15.121:0/1702822128 learned_addr learned my addr 172.21.15.121:0/1702822128 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff85c5d700 1 --1- 172.21.15.121:0/1702822128 >> v1:172.21.15.121:6789/0 conn(0x7eff8009b770 0x7eff800630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47282/0 (socket says 172.21.15.121:47282) 2023-09-29T23:06:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff8645e700 1 --2- 172.21.15.121:0/1702822128 >> v2:172.21.15.123:3300/0 conn(0x7eff80141440 0x7eff800a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 -- 172.21.15.121:0/1702822128 >> v2:172.21.15.123:3300/0 conn(0x7eff80141440 msgr2=0x7eff800a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 --2- 172.21.15.121:0/1702822128 >> v2:172.21.15.123:3300/0 conn(0x7eff80141440 0x7eff800a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 -- 172.21.15.121:0/1702822128 >> v1:172.21.15.121:6789/0 conn(0x7eff8009b770 legacy=0x7eff800630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 -- 172.21.15.121:0/1702822128 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff800a50b0 con 0x7eff800b3cf0 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff8545c700 1 -- 172.21.15.121:0/1702822128 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221347415 0 0) 0x7eff80141810 con 0x7eff8009b770 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff86c5f700 1 --2- 172.21.15.121:0/1702822128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff800b3cf0 0x7eff8014b190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7eff8006bea0 tx=0x7eff78009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdec45eece195aaf server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff8545c700 1 -- 172.21.15.121:0/1702822128 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7800e070 con 0x7eff800b3cf0 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff8545c700 1 -- 172.21.15.121:0/1702822128 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7800ba50 con 0x7eff800b3cf0 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.962+0000 7eff8545c700 1 -- 172.21.15.121:0/1702822128 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff780092e0 con 0x7eff800b3cf0 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 -- 172.21.15.121:0/1702822128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff800b3cf0 msgr2=0x7eff8014b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 --2- 172.21.15.121:0/1702822128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff800b3cf0 0x7eff8014b190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7eff8006bea0 tx=0x7eff78009840 comp rx=0 tx=0).stop 2023-09-29T23:06:37.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 -- 172.21.15.121:0/1702822128 shutdown_connections 2023-09-29T23:06:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 --2- 172.21.15.121:0/1702822128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff800b3cf0 0x7eff8014b190 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 --2- 172.21.15.121:0/1702822128 >> v2:172.21.15.123:3300/0 conn(0x7eff80141440 0x7eff800a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 -- 172.21.15.121:0/1702822128 >> 172.21.15.121:0/1702822128 conn(0x7eff80069e90 msgr2=0x7eff800b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 -- 172.21.15.121:0/1702822128 shutdown_connections 2023-09-29T23:06:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.963+0000 7eff886c2700 1 -- 172.21.15.121:0/1702822128 wait complete. 2023-09-29T23:06:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 Processor -- start 2023-09-29T23:06:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 -- start start 2023-09-29T23:06:37.968 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff8009b770 0x7eff8013c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff800b3cf0 0x7eff8013ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff80141440 0x7eff8009da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8013d380 con 0x7eff8009b770 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8013d610 con 0x7eff80141440 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff886c2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8013cf70 con 0x7eff800b3cf0 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff8645e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff8009b770 0x7eff8013c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff8645e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff8009b770 0x7eff8013c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58160/0 (socket says 172.21.15.121:58160) 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.964+0000 7eff8645e700 1 -- 172.21.15.121:0/717672462 learned_addr learned my addr 172.21.15.121:0/717672462 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff85c5d700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff800b3cf0 0x7eff8013ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff8645e700 1 -- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff800b3cf0 msgr2=0x7eff8013ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff8645e700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff800b3cf0 0x7eff8013ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff8645e700 1 -- 172.21.15.121:0/717672462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff80141440 msgr2=0x7eff8009da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff8645e700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff80141440 0x7eff8009da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff8645e700 1 -- 172.21.15.121:0/717672462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff78009ae0 con 0x7eff8009b770 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff8645e700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff8009b770 0x7eff8013c4f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00d840 tx=0x7eff7c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff767fc700 1 -- 172.21.15.121:0/717672462 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c014620 con 0x7eff8009b770 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff767fc700 1 -- 172.21.15.121:0/717672462 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7c014780 con 0x7eff8009b770 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8006ecb0 con 0x7eff8009b770 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff767fc700 1 -- 172.21.15.121:0/717672462 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c014b40 con 0x7eff8009b770 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.965+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8013ba90 con 0x7eff8009b770 2023-09-29T23:06:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.966+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8006ab60 con 0x7eff8009b770 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.967+0000 7eff767fc700 1 -- 172.21.15.121:0/717672462 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7eff7c01d400 con 0x7eff8009b770 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.967+0000 7eff767fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.967+0000 7eff767fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.967+0000 7eff767fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.967+0000 7eff767fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.967+0000 7eff767fc700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7eff64098780 0x7eff6409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.968+0000 7eff85c5d700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7eff64098780 0x7eff6409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.968+0000 7eff767fc700 1 -- 172.21.15.121:0/717672462 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7eff7c0c1240 con 0x7eff8009b770 2023-09-29T23:06:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.968+0000 7eff85c5d700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7eff64098780 0x7eff6409ac30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff70007fd0 tx=0x7eff70007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:37.972 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:37.971+0000 7eff767fc700 1 -- 172.21.15.121:0/717672462 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff7c0555a0 con 0x7eff8009b770 2023-09-29T23:06:38.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.191+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7eff80071660 con 0x7eff8009b770 2023-09-29T23:06:38.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.191+0000 7eff767fc700 1 -- 172.21.15.121:0/717672462 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7eff7c092370 con 0x7eff8009b770 2023-09-29T23:06:38.191 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:38.192 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:38.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 10 mgrc shutdown 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7eff64098780 msgr2=0x7eff6409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7eff64098780 0x7eff6409ac30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff70007fd0 tx=0x7eff70007d50 comp rx=0 tx=0).stop 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff8009b770 msgr2=0x7eff8013c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff8009b770 0x7eff8013c4f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00d840 tx=0x7eff7c008040 comp rx=0 tx=0).stop 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 shutdown_connections 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7eff64098780 0x7eff6409ac30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff80141440 0x7eff8009da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff8009b770 0x7eff8013c4f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 --2- 172.21.15.121:0/717672462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff800b3cf0 0x7eff8013ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:38.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 >> 172.21.15.121:0/717672462 conn(0x7eff80069e90 msgr2=0x7eff800a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:38.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 shutdown_connections 2023-09-29T23:06:38.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:38.193+0000 7eff886c2700 1 -- 172.21.15.121:0/717672462 wait complete. 2023-09-29T23:06:41.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.333+0000 7fa6c4cd2700 1 Processor -- start 2023-09-29T23:06:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.333+0000 7fa6c4cd2700 1 -- start start 2023-09-29T23:06:41.337 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6c4cd2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00b39c0 0x7fa6c00a5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:41.337 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6c4cd2700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c009dd50 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.337 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6c4cd2700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c009e6d0 con 0x7fa6c00a4cb0 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6c4cd2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c00a63c0 con 0x7fa6c00b39c0 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6bdd9b700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa6c00a5ec0 0x7fa6c00a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47286/0 (socket says 172.21.15.121:47286) 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6bdd9b700 1 -- 172.21.15.121:0/2116564433 learned_addr learned my addr 172.21.15.121:0/2116564433 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6be59c700 1 --2- 172.21.15.121:0/2116564433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00b39c0 0x7fa6c00a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.334+0000 7fa6bd59a700 1 -- 172.21.15.121:0/2116564433 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565677665 0 0) 0x7fa6c009dd50 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6bd59a700 1 -- 172.21.15.121:0/2116564433 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac003570 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6be59c700 1 -- 172.21.15.121:0/2116564433 >> v1:172.21.15.123:6789/0 conn(0x7fa6c00a4cb0 legacy=0x7fa6c0147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6be59c700 1 -- 172.21.15.121:0/2116564433 >> v1:172.21.15.121:6789/0 conn(0x7fa6c00a5ec0 legacy=0x7fa6c00a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6be59c700 1 -- 172.21.15.121:0/2116564433 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c009d7e0 con 0x7fa6c00b39c0 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6be59c700 1 --2- 172.21.15.121:0/2116564433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00b39c0 0x7fa6c00a5980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009900 tx=0x7fa6b0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=524a3005c0064ebe server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6bd59a700 1 -- 172.21.15.121:0/2116564433 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0005c10 con 0x7fa6c00b39c0 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6bd59a700 1 -- 172.21.15.121:0/2116564433 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b0005d70 con 0x7fa6c00b39c0 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6bd59a700 1 -- 172.21.15.121:0/2116564433 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0006260 con 0x7fa6c00b39c0 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/2116564433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00b39c0 msgr2=0x7fa6c00a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.335+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/2116564433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00b39c0 0x7fa6c00a5980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009900 tx=0x7fa6b0009120 comp rx=0 tx=0).stop 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.336+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/2116564433 shutdown_connections 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.336+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/2116564433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00b39c0 0x7fa6c00a5980 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.336+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/2116564433 >> 172.21.15.121:0/2116564433 conn(0x7fa6c0069e90 msgr2=0x7fa6c00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:41.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.336+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/2116564433 shutdown_connections 2023-09-29T23:06:41.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.336+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/2116564433 wait complete. 2023-09-29T23:06:41.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 Processor -- start 2023-09-29T23:06:41.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 -- start start 2023-09-29T23:06:41.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6c00a4cb0 0x7fa6c013e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00a5ec0 0x7fa6c0098f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6c00b39c0 0x7fa6c009a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0098860 con 0x7fa6c00a4cb0 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c006d9b0 con 0x7fa6c00b39c0 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6c4cd2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0098a60 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6be59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6c00a4cb0 0x7fa6c013e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6bdd9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00a5ec0 0x7fa6c0098f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6bdd9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00a5ec0 0x7fa6c0098f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36376/0 (socket says 172.21.15.121:36376) 2023-09-29T23:06:41.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6be59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6c00a4cb0 0x7fa6c013e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58166/0 (socket says 172.21.15.121:58166) 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6bdd9b700 1 -- 172.21.15.121:0/3106532589 learned_addr learned my addr 172.21.15.121:0/3106532589 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.337+0000 7fa6bed9d700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6c00b39c0 0x7fa6c009a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6bdd9b700 1 -- 172.21.15.121:0/3106532589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6c00b39c0 msgr2=0x7fa6c009a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6bdd9b700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6c00b39c0 0x7fa6c009a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6bdd9b700 1 -- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6c00a4cb0 msgr2=0x7fa6c013e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6bdd9b700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6c00a4cb0 0x7fa6c013e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6bdd9b700 1 -- 172.21.15.121:0/3106532589 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00093c0 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6bdd9b700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00a5ec0 0x7fa6c0098f80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800c7f0 tx=0x7fa6a8007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6b67fc700 1 -- 172.21.15.121:0/3106532589 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8005cd0 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6b67fc700 1 -- 172.21.15.121:0/3106532589 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8005e50 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6b67fc700 1 -- 172.21.15.121:0/3106532589 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8007320 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c0095a10 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.338+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c013ede0 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.339+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c00a50f0 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6b67fc700 1 -- 172.21.15.121:0/3106532589 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fa6a801b3b0 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6b67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6b67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6b67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6b67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6b67fc700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa6a00987b0 0x7fa6a009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6be59c700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa6a00987b0 0x7fa6a009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.340+0000 7fa6b67fc700 1 -- 172.21.15.121:0/3106532589 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fa6a80bfab0 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.341+0000 7fa6be59c700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa6a00987b0 0x7fa6a009ac60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009160 tx=0x7fa6b0016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.344+0000 7fa6b67fc700 1 -- 172.21.15.121:0/3106532589 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6a808d1f0 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.564+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa6c006ab60 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.564+0000 7fa6b67fc700 1 -- 172.21.15.121:0/3106532589 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa6a8017080 con 0x7fa6c00a5ec0 2023-09-29T23:06:41.566 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:41.566 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.566+0000 7fa6c4cd2700 10 mgrc shutdown 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.566+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa6a00987b0 msgr2=0x7fa6a009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.566+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa6a00987b0 0x7fa6a009ac60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009160 tx=0x7fa6b0016000 comp rx=0 tx=0).stop 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.566+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00a5ec0 msgr2=0x7fa6c0098f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.566+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00a5ec0 0x7fa6c0098f80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800c7f0 tx=0x7fa6a8007e00 comp rx=0 tx=0).stop 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 shutdown_connections 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa6a00987b0 0x7fa6a009ac60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6c00b39c0 0x7fa6c009a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6c00a4cb0 0x7fa6c013e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 --2- 172.21.15.121:0/3106532589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6c00a5ec0 0x7fa6c0098f80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:41.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 >> 172.21.15.121:0/3106532589 conn(0x7fa6c0069e90 msgr2=0x7fa6c00b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:41.569 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 shutdown_connections 2023-09-29T23:06:41.569 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:41.567+0000 7fa6c4cd2700 1 -- 172.21.15.121:0/3106532589 wait complete. 2023-09-29T23:06:44.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:44.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.708+0000 7f0d5e839700 1 Processor -- start 2023-09-29T23:06:44.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d5e839700 1 -- start start 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d5e839700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d5814c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d5e839700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d58141810 con 0x7f0d58141440 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d5e839700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5809a7f0 con 0x7f0d5809b770 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d5e839700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d580a4df0 con 0x7f0d580b3cf0 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d5cdd6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d5814c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d57fff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0d58141440 0x7f0d580a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47292/0 (socket says 172.21.15.121:47292) 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.709+0000 7f0d57fff700 1 -- 172.21.15.121:0/2760805807 learned_addr learned my addr 172.21.15.121:0/2760805807 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d56ffd700 1 -- 172.21.15.121:0/2760805807 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399972877 0 0) 0x7f0d58141810 con 0x7f0d58141440 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d56ffd700 1 -- 172.21.15.121:0/2760805807 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d40003570 con 0x7f0d58141440 2023-09-29T23:06:44.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d5cdd6700 1 -- 172.21.15.121:0/2760805807 >> v1:172.21.15.123:6789/0 conn(0x7f0d5809b770 legacy=0x7f0d580630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d5cdd6700 1 -- 172.21.15.121:0/2760805807 >> v1:172.21.15.121:6789/0 conn(0x7f0d58141440 legacy=0x7f0d580a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d5cdd6700 1 -- 172.21.15.121:0/2760805807 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d580a50b0 con 0x7f0d580b3cf0 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d5cdd6700 1 --2- 172.21.15.121:0/2760805807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d5814c1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009e90 tx=0x7f0d50009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f7816ec0c730e64 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d56ffd700 1 -- 172.21.15.121:0/2760805807 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5000e070 con 0x7f0d580b3cf0 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d56ffd700 1 -- 172.21.15.121:0/2760805807 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5000bc30 con 0x7f0d580b3cf0 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d56ffd700 1 -- 172.21.15.121:0/2760805807 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50012430 con 0x7f0d580b3cf0 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d5e839700 1 -- 172.21.15.121:0/2760805807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 msgr2=0x7f0d5814c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.710+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2760805807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d5814c1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009e90 tx=0x7f0d50009780 comp rx=0 tx=0).stop 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.711+0000 7f0d5e839700 1 -- 172.21.15.121:0/2760805807 shutdown_connections 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.711+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2760805807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d5814c1a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:44.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.711+0000 7f0d5e839700 1 -- 172.21.15.121:0/2760805807 >> 172.21.15.121:0/2760805807 conn(0x7f0d58069e90 msgr2=0x7f0d580b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:44.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.711+0000 7f0d5e839700 1 -- 172.21.15.121:0/2760805807 shutdown_connections 2023-09-29T23:06:44.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.711+0000 7f0d5e839700 1 -- 172.21.15.121:0/2760805807 wait complete. 2023-09-29T23:06:44.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.711+0000 7f0d5e839700 1 Processor -- start 2023-09-29T23:06:44.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5e839700 1 -- start start 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5e839700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d5809b770 0x7f0d581446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5e839700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d58145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5e839700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d58141440 0x7f0d58145f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5e839700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d581438c0 con 0x7f0d5809b770 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5e839700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d581464b0 con 0x7f0d58141440 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5e839700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d58143ce0 con 0x7f0d580b3cf0 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d57fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d5809b770 0x7f0d581446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d577fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d58145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d57fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d5809b770 0x7f0d581446a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58168/0 (socket says 172.21.15.121:58168) 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d57fff700 1 -- 172.21.15.121:0/2253435680 learned_addr learned my addr 172.21.15.121:0/2253435680 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:44.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.712+0000 7f0d5cdd6700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d58141440 0x7f0d58145f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d57fff700 1 -- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 msgr2=0x7f0d58145a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d57fff700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d58145a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d57fff700 1 -- 172.21.15.121:0/2253435680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d58141440 msgr2=0x7f0d58145f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d57fff700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d58141440 0x7f0d58145f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d57fff700 1 -- 172.21.15.121:0/2253435680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50009a20 con 0x7f0d5809b770 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d577fe700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d58145a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d57fff700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d5809b770 0x7f0d581446a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0d4800c790 tx=0x7f0d48007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d3ffff700 1 -- 172.21.15.121:0/2253435680 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480072e0 con 0x7f0d5809b770 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d3ffff700 1 -- 172.21.15.121:0/2253435680 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d48007440 con 0x7f0d5809b770 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d3ffff700 1 -- 172.21.15.121:0/2253435680 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480057f0 con 0x7f0d5809b770 2023-09-29T23:06:44.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d581466e0 con 0x7f0d5809b770 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.713+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d581469d0 con 0x7f0d5809b770 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.714+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5806ab60 con 0x7f0d5809b770 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.715+0000 7f0d3ffff700 1 -- 172.21.15.121:0/2253435680 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f0d48005ba0 con 0x7f0d5809b770 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.715+0000 7f0d3ffff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.715+0000 7f0d3ffff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.715+0000 7f0d3ffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.715+0000 7f0d3ffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.715+0000 7f0d3ffff700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0d440987b0 0x7f0d4409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.716+0000 7f0d577fe700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0d440987b0 0x7f0d4409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.716+0000 7f0d3ffff700 1 -- 172.21.15.121:0/2253435680 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f0d480bf640 con 0x7f0d5809b770 2023-09-29T23:06:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.716+0000 7f0d577fe700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0d440987b0 0x7f0d4409ac60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0d58146f10 tx=0x7f0d4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:44.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.720+0000 7f0d3ffff700 1 -- 172.21.15.121:0/2253435680 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0d4804b5a0 con 0x7f0d5809b770 2023-09-29T23:06:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.942+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0d58141c50 con 0x7f0d5809b770 2023-09-29T23:06:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.942+0000 7f0d3ffff700 1 -- 172.21.15.121:0/2253435680 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0d48017100 con 0x7f0d5809b770 2023-09-29T23:06:44.943 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:44.943 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 10 mgrc shutdown 2023-09-29T23:06:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0d440987b0 msgr2=0x7f0d4409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0d440987b0 0x7f0d4409ac60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0d58146f10 tx=0x7f0d4c009040 comp rx=0 tx=0).stop 2023-09-29T23:06:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d5809b770 msgr2=0x7f0d581446a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d5809b770 0x7f0d581446a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0d4800c790 tx=0x7f0d48007c80 comp rx=0 tx=0).stop 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 shutdown_connections 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0d440987b0 0x7f0d4409ac60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d58141440 0x7f0d58145f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d5809b770 0x7f0d581446a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 --2- 172.21.15.121:0/2253435680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d580b3cf0 0x7f0d58145a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 >> 172.21.15.121:0/2253435680 conn(0x7f0d58069e90 msgr2=0x7f0d580a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 shutdown_connections 2023-09-29T23:06:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:44.944+0000 7f0d5e839700 1 -- 172.21.15.121:0/2253435680 wait complete. 2023-09-29T23:06:47.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.085+0000 7f70df6d9700 1 Processor -- start 2023-09-29T23:06:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.085+0000 7f70df6d9700 1 -- start start 2023-09-29T23:06:48.089 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70df6d9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70d80a6950 0x7f70d80a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:48.089 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70df6d9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d813d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:48.089 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70df6d9700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d80a85e0 con 0x7f70d80a4480 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70df6d9700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d80a6d20 con 0x7f70d80a6950 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70df6d9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d80a91a0 con 0x7f70d80a73b0 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dd475700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70d80a6950 0x7f70d80a8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d813d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d813d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:49668/0 (socket says 172.21.15.121:49668) 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dd475700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70d80a6950 0x7f70d80a8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49888/0 (socket says 172.21.15.121:49888) 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 -- 172.21.15.121:0/450844640 learned_addr learned my addr 172.21.15.121:0/450844640 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 -- 172.21.15.121:0/450844640 >> v2:172.21.15.121:3300/0 conn(0x7f70d80a6950 msgr2=0x7f70d80a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 --2- 172.21.15.121:0/450844640 >> v2:172.21.15.121:3300/0 conn(0x7f70d80a6950 0x7f70d80a8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 -- 172.21.15.121:0/450844640 >> v1:172.21.15.123:6789/0 conn(0x7f70d80a4480 legacy=0x7f70d80a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 -- 172.21.15.121:0/450844640 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d80a9440 con 0x7f70d80a73b0 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dd475700 1 --2- 172.21.15.121:0/450844640 >> v2:172.21.15.121:3300/0 conn(0x7f70d80a6950 0x7f70d80a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.086+0000 7f70dcc74700 1 --2- 172.21.15.121:0/450844640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d813d840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70d40099b0 tx=0x7f70d4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b6060f29b7b7ec server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70cffff700 1 -- 172.21.15.121:0/450844640 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4009ea0 con 0x7f70d80a73b0 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70cffff700 1 -- 172.21.15.121:0/450844640 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d400b8f0 con 0x7f70d80a73b0 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70cffff700 1 -- 172.21.15.121:0/450844640 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400bc90 con 0x7f70d80a73b0 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 -- 172.21.15.121:0/450844640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 msgr2=0x7f70d813d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 --2- 172.21.15.121:0/450844640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d813d840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70d40099b0 tx=0x7f70d4009250 comp rx=0 tx=0).stop 2023-09-29T23:06:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 -- 172.21.15.121:0/450844640 shutdown_connections 2023-09-29T23:06:48.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 --2- 172.21.15.121:0/450844640 >> v2:172.21.15.121:3300/0 conn(0x7f70d80a6950 0x7f70d80a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 --2- 172.21.15.121:0/450844640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d813d840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 -- 172.21.15.121:0/450844640 >> 172.21.15.121:0/450844640 conn(0x7f70d806c010 msgr2=0x7f70d806ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:48.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 -- 172.21.15.121:0/450844640 shutdown_connections 2023-09-29T23:06:48.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.087+0000 7f70df6d9700 1 -- 172.21.15.121:0/450844640 wait complete. 2023-09-29T23:06:48.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.088+0000 7f70df6d9700 1 Processor -- start 2023-09-29T23:06:48.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.088+0000 7f70df6d9700 1 -- start start 2023-09-29T23:06:48.095 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70df6d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d80a4480 0x7f70d806e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:48.095 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70df6d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d80a6950 0x7f70d8137600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:48.095 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70df6d9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d80b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:48.095 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70df6d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8137ec0 con 0x7f70d80a6950 2023-09-29T23:06:48.095 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70df6d9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d81380c0 con 0x7f70d80a4480 2023-09-29T23:06:48.095 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70df6d9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8137b40 con 0x7f70d80a73b0 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70dcc74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d80a6950 0x7f70d8137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70ddc76700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d80b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70dcc74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d80a6950 0x7f70d8137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49894/0 (socket says 172.21.15.121:49894) 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70dcc74700 1 -- 172.21.15.121:0/339904871 learned_addr learned my addr 172.21.15.121:0/339904871 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.089+0000 7f70dd475700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d80a4480 0x7f70d806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70dcc74700 1 -- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 msgr2=0x7f70d80b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70dcc74700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d80b0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70dcc74700 1 -- 172.21.15.121:0/339904871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d80a4480 msgr2=0x7f70d806e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70dcc74700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d80a4480 0x7f70d806e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70dcc74700 1 -- 172.21.15.121:0/339904871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c800b6d0 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70ddc76700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d80b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70dcc74700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d80a6950 0x7f70d8137600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f70d806cb80 tx=0x7f70d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70cd7fa700 1 -- 172.21.15.121:0/339904871 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400ba60 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70cd7fa700 1 -- 172.21.15.121:0/339904871 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d400bbe0 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d40094f0 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70cd7fa700 1 -- 172.21.15.121:0/339904871 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4026680 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.090+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d80aef00 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.091+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d8071660 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70cd7fa700 1 -- 172.21.15.121:0/339904871 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f70d4023030 con 0x7f70d80a6950 2023-09-29T23:06:48.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70cd7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:48.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70cd7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:48.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70cd7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:48.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70cd7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:48.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70cd7fa700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f70bc098780 0x7f70bc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:48.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70dd475700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f70bc098780 0x7f70bc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:48.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.092+0000 7f70cd7fa700 1 -- 172.21.15.121:0/339904871 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f70d40c1f20 con 0x7f70d80a6950 2023-09-29T23:06:48.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.093+0000 7f70dd475700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f70bc098780 0x7f70bc09ac30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f70c80114a0 tx=0x7f70c800b430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:48.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.096+0000 7f70cd7fa700 1 -- 172.21.15.121:0/339904871 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f70d408f660 con 0x7f70d80a6950 2023-09-29T23:06:48.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.318+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70d80a9790 con 0x7f70d80a6950 2023-09-29T23:06:48.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.318+0000 7f70cd7fa700 1 -- 172.21.15.121:0/339904871 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f70d4092c80 con 0x7f70d80a6950 2023-09-29T23:06:48.320 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:48.321 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 10 mgrc shutdown 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f70bc098780 msgr2=0x7f70bc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f70bc098780 0x7f70bc09ac30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f70c80114a0 tx=0x7f70c800b430 comp rx=0 tx=0).stop 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d80a6950 msgr2=0x7f70d8137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d80a6950 0x7f70d8137600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f70d806cb80 tx=0x7f70d4017000 comp rx=0 tx=0).stop 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 shutdown_connections 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f70bc098780 0x7f70bc09ac30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70d80a73b0 0x7f70d80b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d80a4480 0x7f70d806e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 --2- 172.21.15.121:0/339904871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d80a6950 0x7f70d8137600 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:48.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 >> 172.21.15.121:0/339904871 conn(0x7f70d806c010 msgr2=0x7f70d813f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:48.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 shutdown_connections 2023-09-29T23:06:48.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:48.320+0000 7f70df6d9700 1 -- 172.21.15.121:0/339904871 wait complete. 2023-09-29T23:06:51.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.461+0000 7fdc231bc700 1 Processor -- start 2023-09-29T23:06:51.462 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.461+0000 7fdc231bc700 1 -- start start 2023-09-29T23:06:51.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc231bc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c09d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:51.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc231bc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdc1c141c60 0x7fdc1c142050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:51.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc231bc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c06aa00 con 0x7fdc1c134050 2023-09-29T23:06:51.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc231bc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c134bb0 con 0x7fdc1c141c60 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc231bc700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c06bbd0 con 0x7fdc1c06a630 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c09d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c09d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:49688/0 (socket says 172.21.15.121:49688) 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 -- 172.21.15.121:0/3947430866 learned_addr learned my addr 172.21.15.121:0/3947430866 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc1bfff700 1 --2- 172.21.15.121:0/3947430866 >> v2:172.21.15.123:3300/0 conn(0x7fdc1c141c60 0x7fdc1c142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 -- 172.21.15.121:0/3947430866 >> v2:172.21.15.123:3300/0 conn(0x7fdc1c141c60 msgr2=0x7fdc1c142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 --2- 172.21.15.121:0/3947430866 >> v2:172.21.15.123:3300/0 conn(0x7fdc1c141c60 0x7fdc1c142050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 -- 172.21.15.121:0/3947430866 >> v1:172.21.15.123:6789/0 conn(0x7fdc1c134050 legacy=0x7fdc1c1344a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 -- 172.21.15.121:0/3947430866 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c1417c0 con 0x7fdc1c06a630 2023-09-29T23:06:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc20f58700 1 --2- 172.21.15.121:0/3947430866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c09d180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdc100099b0 tx=0x7fdc10009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98d7a6068877c81 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc1b7fe700 1 -- 172.21.15.121:0/3947430866 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000b820 con 0x7fdc1c06a630 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc1b7fe700 1 -- 172.21.15.121:0/3947430866 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1000b980 con 0x7fdc1c06a630 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.462+0000 7fdc1b7fe700 1 -- 172.21.15.121:0/3947430866 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000bd20 con 0x7fdc1c06a630 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 -- 172.21.15.121:0/3947430866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 msgr2=0x7fdc1c09d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3947430866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c09d180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdc100099b0 tx=0x7fdc10009250 comp rx=0 tx=0).stop 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 -- 172.21.15.121:0/3947430866 shutdown_connections 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3947430866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c09d180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3947430866 >> v2:172.21.15.123:3300/0 conn(0x7fdc1c141c60 0x7fdc1c142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 -- 172.21.15.121:0/3947430866 >> 172.21.15.121:0/3947430866 conn(0x7fdc1c068fd0 msgr2=0x7fdc1c069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 -- 172.21.15.121:0/3947430866 shutdown_connections 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.463+0000 7fdc231bc700 1 -- 172.21.15.121:0/3947430866 wait complete. 2023-09-29T23:06:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.464+0000 7fdc231bc700 1 Processor -- start 2023-09-29T23:06:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.464+0000 7fdc231bc700 1 -- start start 2023-09-29T23:06:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c0a2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c134050 0x7fdc1c097570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1c141c60 0x7fdc1c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c096480 con 0x7fdc1c141c60 2023-09-29T23:06:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c097ab0 con 0x7fdc1c134050 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0b4350 con 0x7fdc1c06a630 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c0a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c0a2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:49698/0 (socket says 172.21.15.121:49698) 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 -- 172.21.15.121:0/3845060489 learned_addr learned my addr 172.21.15.121:0/3845060489 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc21759700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1c141c60 0x7fdc1c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc1bfff700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c134050 0x7fdc1c097570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 -- 172.21.15.121:0/3845060489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c134050 msgr2=0x7fdc1c097570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c134050 0x7fdc1c097570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 -- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1c141c60 msgr2=0x7fdc1c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:51.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1c141c60 0x7fdc1c0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 -- 172.21.15.121:0/3845060489 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc100094f0 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc21759700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1c141c60 0x7fdc1c0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc20f58700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c0a2890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc100094c0 tx=0x7fdc10016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc18ff9700 1 -- 172.21.15.121:0/3845060489 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10011bf0 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc18ff9700 1 -- 172.21.15.121:0/3845060489 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc10020070 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c097070 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc18ff9700 1 -- 172.21.15.121:0/3845060489 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1001b450 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.465+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c0b29d0 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.466+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1c0973c0 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.467+0000 7fdc18ff9700 1 -- 172.21.15.121:0/3845060489 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fdc10022030 con 0x7fdc1c06a630 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.467+0000 7fdc18ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.468+0000 7fdc18ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.468+0000 7fdc18ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.468+0000 7fdc18ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.468+0000 7fdc18ff9700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdc00098780 0x7fdc0009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.468+0000 7fdc1bfff700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdc00098780 0x7fdc0009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.468+0000 7fdc18ff9700 1 -- 172.21.15.121:0/3845060489 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fdc100c21c0 con 0x7fdc1c06a630 2023-09-29T23:06:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.468+0000 7fdc1bfff700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdc00098780 0x7fdc0009ac30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c09cc40 tx=0x7fdc08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:51.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.472+0000 7fdc18ff9700 1 -- 172.21.15.121:0/3845060489 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc1008f900 con 0x7fdc1c06a630 2023-09-29T23:06:51.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.694+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc1c0973c0 con 0x7fdc1c06a630 2023-09-29T23:06:51.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.694+0000 7fdc18ff9700 1 -- 172.21.15.121:0/3845060489 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdc10092f20 con 0x7fdc1c06a630 2023-09-29T23:06:51.695 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:51.695 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.696+0000 7fdc231bc700 10 mgrc shutdown 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.696+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdc00098780 msgr2=0x7fdc0009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.696+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdc00098780 0x7fdc0009ac30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c09cc40 tx=0x7fdc08008040 comp rx=0 tx=0).stop 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 msgr2=0x7fdc1c0a2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c0a2890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc100094c0 tx=0x7fdc10016000 comp rx=0 tx=0).stop 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 shutdown_connections 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdc00098780 0x7fdc0009ac30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1c141c60 0x7fdc1c0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc1c06a630 0x7fdc1c0a2890 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.699 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 --2- 172.21.15.121:0/3845060489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc1c134050 0x7fdc1c097570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:51.699 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 >> 172.21.15.121:0/3845060489 conn(0x7fdc1c068fd0 msgr2=0x7fdc1c070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:51.699 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 shutdown_connections 2023-09-29T23:06:51.699 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:51.697+0000 7fdc231bc700 1 -- 172.21.15.121:0/3845060489 wait complete. 2023-09-29T23:06:54.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:54.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.837+0000 7febdd129700 1 Processor -- start 2023-09-29T23:06:54.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.837+0000 7febdd129700 1 -- start start 2023-09-29T23:06:54.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febdd129700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7febd81343a0 0x7febd81347f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:54.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febdd129700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd80b57b0 con 0x7febd8141f70 2023-09-29T23:06:54.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febdd129700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd8141ae0 con 0x7febd80b53e0 2023-09-29T23:06:54.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febdd129700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd80a7140 con 0x7febd81343a0 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febd759e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7febd81343a0 0x7febd81347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febd759e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7febd81343a0 0x7febd81347f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49910/0 (socket says 172.21.15.121:49910) 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febd659c700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7febd8141f70 0x7febd81423b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58594/0 (socket says 172.21.15.121:58594) 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febd759e700 1 -- 172.21.15.121:0/3182555265 learned_addr learned my addr 172.21.15.121:0/3182555265 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febd759e700 1 -- 172.21.15.121:0/3182555265 >> v1:172.21.15.123:6789/0 conn(0x7febd80b53e0 legacy=0x7febd809e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febd759e700 1 -- 172.21.15.121:0/3182555265 >> v1:172.21.15.121:6789/0 conn(0x7febd8141f70 legacy=0x7febd81423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.838+0000 7febd759e700 1 -- 172.21.15.121:0/3182555265 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd809e930 con 0x7febd81343a0 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febd759e700 1 --2- 172.21.15.121:0/3182555265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd81347f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7febd0009f20 tx=0x7febd0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac0ed8e15550433 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febd5d9b700 1 -- 172.21.15.121:0/3182555265 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd000e070 con 0x7febd81343a0 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febd5d9b700 1 -- 172.21.15.121:0/3182555265 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd000ba40 con 0x7febd81343a0 2023-09-29T23:06:54.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febd5d9b700 1 -- 172.21.15.121:0/3182555265 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd00092e0 con 0x7febd81343a0 2023-09-29T23:06:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febdd129700 1 -- 172.21.15.121:0/3182555265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 msgr2=0x7febd81347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febdd129700 1 --2- 172.21.15.121:0/3182555265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd81347f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7febd0009f20 tx=0x7febd0009840 comp rx=0 tx=0).stop 2023-09-29T23:06:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febdd129700 1 -- 172.21.15.121:0/3182555265 shutdown_connections 2023-09-29T23:06:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febdd129700 1 --2- 172.21.15.121:0/3182555265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd81347f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febdd129700 1 -- 172.21.15.121:0/3182555265 >> 172.21.15.121:0/3182555265 conn(0x7febd8067f10 msgr2=0x7febd80b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febdd129700 1 -- 172.21.15.121:0/3182555265 shutdown_connections 2023-09-29T23:06:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.839+0000 7febdd129700 1 -- 172.21.15.121:0/3182555265 wait complete. 2023-09-29T23:06:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.840+0000 7febdd129700 1 Processor -- start 2023-09-29T23:06:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.840+0000 7febdd129700 1 -- start start 2023-09-29T23:06:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febdd129700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b53e0 0x7febd80a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febdd129700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd80b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febdd129700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febd8141f70 0x7febd8069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febdd129700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd80b0b60 con 0x7febd81343a0 2023-09-29T23:06:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febdd129700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd80b0e20 con 0x7febd8141f70 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febdd129700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd80b10e0 con 0x7febd80b53e0 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd80b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd80b0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49914/0 (socket says 172.21.15.121:49914) 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 -- 172.21.15.121:0/2366883067 learned_addr learned my addr 172.21.15.121:0/2366883067 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd6d9d700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b53e0 0x7febd80a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd759e700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febd8141f70 0x7febd8069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 -- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b53e0 msgr2=0x7febd80a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b53e0 0x7febd80a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 -- 172.21.15.121:0/2366883067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febd8141f70 msgr2=0x7febd8069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:54.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febd8141f70 0x7febd8069560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 -- 172.21.15.121:0/2366883067 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd0009ae0 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.841+0000 7febd659c700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd80b0510 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7febc800c560 tx=0x7febc8007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.842+0000 7febceffd700 1 -- 172.21.15.121:0/2366883067 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8005a70 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.842+0000 7febceffd700 1 -- 172.21.15.121:0/2366883067 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc8005bd0 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.842+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd81416c0 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.842+0000 7febceffd700 1 -- 172.21.15.121:0/2366883067 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc80070d0 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.842+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd8069aa0 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.842+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd80a40f0 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febceffd700 1 -- 172.21.15.121:0/2366883067 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7febc8007440 con 0x7febd81343a0 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febceffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febceffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:54.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febceffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:54.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febceffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:54.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febceffd700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7febb8098780 0x7febb809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:54.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febd6d9d700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7febb8098780 0x7febb809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:54.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.844+0000 7febceffd700 1 -- 172.21.15.121:0/2366883067 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7febc80bfba0 con 0x7febd81343a0 2023-09-29T23:06:54.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.845+0000 7febd6d9d700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7febb8098780 0x7febb809ac30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7febc0003a30 tx=0x7febc0007ea0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:54.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:54.847+0000 7febceffd700 1 -- 172.21.15.121:0/2366883067 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7febc808d240 con 0x7febd81343a0 2023-09-29T23:06:55.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.069+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7febd8071660 con 0x7febd81343a0 2023-09-29T23:06:55.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.069+0000 7febceffd700 1 -- 172.21.15.121:0/2366883067 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7febc8017100 con 0x7febd81343a0 2023-09-29T23:06:55.071 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:55.071 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:55.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 10 mgrc shutdown 2023-09-29T23:06:55.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7febb8098780 msgr2=0x7febb809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:55.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7febb8098780 0x7febb809ac30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7febc0003a30 tx=0x7febc0007ea0 comp rx=0 tx=0).stop 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 msgr2=0x7febd80b0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd80b0510 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7febc800c560 tx=0x7febc8007f60 comp rx=0 tx=0).stop 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 shutdown_connections 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7febb8098780 0x7febb809ac30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febd8141f70 0x7febd8069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b53e0 0x7febd80a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 --2- 172.21.15.121:0/2366883067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd81343a0 0x7febd80b0510 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 >> 172.21.15.121:0/2366883067 conn(0x7febd8067f10 msgr2=0x7febd80a0490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 shutdown_connections 2023-09-29T23:06:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:55.071+0000 7febdd129700 1 -- 172.21.15.121:0/2366883067 wait complete. 2023-09-29T23:06:58.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:06:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.212+0000 7f6aa9041700 1 Processor -- start 2023-09-29T23:06:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.212+0000 7f6aa9041700 1 -- start start 2023-09-29T23:06:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.212+0000 7f6aa9041700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6aa40b3220 0x7f6aa40b3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.212+0000 7f6aa9041700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40abdd0 0x7f6aa40ac200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.212+0000 7f6aa9041700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa41445d0 con 0x7f6aa4143940 2023-09-29T23:06:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.212+0000 7f6aa9041700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa40a6920 con 0x7f6aa40b3220 2023-09-29T23:06:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.212+0000 7f6aa9041700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa40b3ec0 con 0x7f6aa40abdd0 2023-09-29T23:06:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40abdd0 0x7f6aa40ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa2d9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6aa40b3220 0x7f6aa40b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40abdd0 0x7f6aa40ac200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:57710/0 (socket says 172.21.15.121:57710) 2023-09-29T23:06:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa2d9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6aa40b3220 0x7f6aa40b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53450/0 (socket says 172.21.15.121:53450) 2023-09-29T23:06:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 -- 172.21.15.121:0/2468919963 learned_addr learned my addr 172.21.15.121:0/2468919963 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 -- 172.21.15.121:0/2468919963 >> v2:172.21.15.121:3300/0 conn(0x7f6aa40b3220 msgr2=0x7f6aa40b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:06:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 --2- 172.21.15.121:0/2468919963 >> v2:172.21.15.121:3300/0 conn(0x7f6aa40b3220 0x7f6aa40b3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 -- 172.21.15.121:0/2468919963 >> v1:172.21.15.123:6789/0 conn(0x7f6aa4143940 legacy=0x7f6aa4143d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 -- 172.21.15.121:0/2468919963 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa4136a40 con 0x7f6aa40abdd0 2023-09-29T23:06:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa2d9d700 1 --2- 172.21.15.121:0/2468919963 >> v2:172.21.15.121:3300/0 conn(0x7f6aa40b3220 0x7f6aa40b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa259c700 1 --2- 172.21.15.121:0/2468919963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40abdd0 0x7f6aa40ac200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a980099b0 tx=0x7f6a98009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc19bcd08b1213e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa1d9b700 1 -- 172.21.15.121:0/2468919963 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98009ea0 con 0x7f6aa40abdd0 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa1d9b700 1 -- 172.21.15.121:0/2468919963 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9800b860 con 0x7f6aa40abdd0 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.213+0000 7f6aa1d9b700 1 -- 172.21.15.121:0/2468919963 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9800bc00 con 0x7f6aa40abdd0 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 -- 172.21.15.121:0/2468919963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40abdd0 msgr2=0x7f6aa40ac200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 --2- 172.21.15.121:0/2468919963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40abdd0 0x7f6aa40ac200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a980099b0 tx=0x7f6a98009250 comp rx=0 tx=0).stop 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 -- 172.21.15.121:0/2468919963 shutdown_connections 2023-09-29T23:06:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 --2- 172.21.15.121:0/2468919963 >> v2:172.21.15.121:3300/0 conn(0x7f6aa40b3220 0x7f6aa40b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 --2- 172.21.15.121:0/2468919963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40abdd0 0x7f6aa40ac200 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 -- 172.21.15.121:0/2468919963 >> 172.21.15.121:0/2468919963 conn(0x7f6aa406c010 msgr2=0x7f6aa406ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 -- 172.21.15.121:0/2468919963 shutdown_connections 2023-09-29T23:06:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.214+0000 7f6aa9041700 1 -- 172.21.15.121:0/2468919963 wait complete. 2023-09-29T23:06:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 Processor -- start 2023-09-29T23:06:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 -- start start 2023-09-29T23:06:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa40abdd0 0x7f6aa40ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40b3220 0x7f6aa40acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa4143940 0x7f6aa40af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa412a670 con 0x7f6aa40abdd0 2023-09-29T23:06:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa40ad780 con 0x7f6aa4143940 2023-09-29T23:06:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa9041700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa40ada20 con 0x7f6aa40b3220 2023-09-29T23:06:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.215+0000 7f6aa2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa40abdd0 0x7f6aa40ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa40abdd0 0x7f6aa40ae3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53460/0 (socket says 172.21.15.121:53460) 2023-09-29T23:06:58.225 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 -- 172.21.15.121:0/1299558045 learned_addr learned my addr 172.21.15.121:0/1299558045 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:06:58.225 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40b3220 0x7f6aa40acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:58.225 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa359e700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa4143940 0x7f6aa40af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:58.225 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 -- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40b3220 msgr2=0x7f6aa40acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:58.225 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40b3220 0x7f6aa40acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.225 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 -- 172.21.15.121:0/1299558045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa4143940 msgr2=0x7f6aa40af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:58.226 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa4143940 0x7f6aa40af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.226 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 -- 172.21.15.121:0/1299558045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a94009420 con 0x7f6aa40abdd0 2023-09-29T23:06:58.226 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa2d9d700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa40abdd0 0x7f6aa40ae3d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400ec80 tx=0x7f6a940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:58.226 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6a8affd700 1 -- 172.21.15.121:0/1299558045 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9400c920 con 0x7f6aa40abdd0 2023-09-29T23:06:58.226 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6a8affd700 1 -- 172.21.15.121:0/1299558045 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9400ca80 con 0x7f6aa40abdd0 2023-09-29T23:06:58.226 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a980094f0 con 0x7f6aa40abdd0 2023-09-29T23:06:58.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6a8affd700 1 -- 172.21.15.121:0/1299558045 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9400ce00 con 0x7f6aa40abdd0 2023-09-29T23:06:58.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.216+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa4131560 con 0x7f6aa40abdd0 2023-09-29T23:06:58.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.217+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa4071660 con 0x7f6aa40abdd0 2023-09-29T23:06:58.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.218+0000 7f6a8affd700 1 -- 172.21.15.121:0/1299558045 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6a94013680 con 0x7f6aa40abdd0 2023-09-29T23:06:58.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.218+0000 7f6a8affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:06:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.219+0000 7f6a8affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:06:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.219+0000 7f6a8affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.219+0000 7f6a8affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:06:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.219+0000 7f6a8affd700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6a90098780 0x7f6a9009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:06:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.219+0000 7f6aa259c700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6a90098780 0x7f6a9009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:06:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.219+0000 7f6a8affd700 1 -- 172.21.15.121:0/1299558045 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f6a940c0b80 con 0x7f6aa40abdd0 2023-09-29T23:06:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.219+0000 7f6aa259c700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6a90098780 0x7f6a9009ac30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6a980099b0 tx=0x7f6a98017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:06:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.223+0000 7f6a8affd700 1 -- 172.21.15.121:0/1299558045 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a9408e220 con 0x7f6aa40abdd0 2023-09-29T23:06:58.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.444+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6aa40a4c40 con 0x7f6aa40abdd0 2023-09-29T23:06:58.446 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.445+0000 7f6a8affd700 1 -- 172.21.15.121:0/1299558045 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6a94091840 con 0x7f6aa40abdd0 2023-09-29T23:06:58.446 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:06:58.446 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:06:58.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 10 mgrc shutdown 2023-09-29T23:06:58.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6a90098780 msgr2=0x7f6a9009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:58.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6a90098780 0x7f6a9009ac30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6a980099b0 tx=0x7f6a98017000 comp rx=0 tx=0).stop 2023-09-29T23:06:58.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa40abdd0 msgr2=0x7f6aa40ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa40abdd0 0x7f6aa40ae3d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400ec80 tx=0x7f6a940091a0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 shutdown_connections 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6a90098780 0x7f6a9009ac30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aa4143940 0x7f6aa40af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa40abdd0 0x7f6aa40ae3d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 --2- 172.21.15.121:0/1299558045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa40b3220 0x7f6aa40acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 >> 172.21.15.121:0/1299558045 conn(0x7f6aa406c010 msgr2=0x7f6aa4133270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 shutdown_connections 2023-09-29T23:06:58.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:06:58.447+0000 7f6aa9041700 1 -- 172.21.15.121:0/1299558045 wait complete. 2023-09-29T23:07:01.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f3419177700 1 Processor -- start 2023-09-29T23:07:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f3419177700 1 -- start start 2023-09-29T23:07:01.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f3419177700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34141438a0 0x7f3414143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:01.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f3419177700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3414144530 0x7f3414140970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:01.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f3419177700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341406aa00 con 0x7f341406a630 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f3419177700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34141406b0 con 0x7f34141438a0 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f3419177700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34141402b0 con 0x7f3414144530 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.590+0000 7f341259c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34141438a0 0x7f3414143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341259c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34141438a0 0x7f3414143ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53472/0 (socket says 172.21.15.121:53472) 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341259c700 1 -- 172.21.15.121:0/176911385 learned_addr learned my addr 172.21.15.121:0/176911385 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341259c700 1 -- 172.21.15.121:0/176911385 >> v2:172.21.15.123:3300/0 conn(0x7f3414144530 msgr2=0x7f3414140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341359e700 1 --2- 172.21.15.121:0/176911385 >> v2:172.21.15.123:3300/0 conn(0x7f3414144530 0x7f3414140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341259c700 1 --2- 172.21.15.121:0/176911385 >> v2:172.21.15.123:3300/0 conn(0x7f3414144530 0x7f3414140970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341259c700 1 -- 172.21.15.121:0/176911385 >> v1:172.21.15.123:6789/0 conn(0x7f341406a630 legacy=0x7f341406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341259c700 1 -- 172.21.15.121:0/176911385 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3414140420 con 0x7f34141438a0 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f341259c700 1 --2- 172.21.15.121:0/176911385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414143ff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f341406e4f0 tx=0x7f3408009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291daf6c1359a6b1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f3411d9b700 1 -- 172.21.15.121:0/176911385 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340800b820 con 0x7f34141438a0 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f3411d9b700 1 -- 172.21.15.121:0/176911385 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340800b980 con 0x7f34141438a0 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.591+0000 7f3411d9b700 1 -- 172.21.15.121:0/176911385 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340800bd20 con 0x7f34141438a0 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 -- 172.21.15.121:0/176911385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 msgr2=0x7f3414143ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 --2- 172.21.15.121:0/176911385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414143ff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f341406e4f0 tx=0x7f3408009250 comp rx=0 tx=0).stop 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 -- 172.21.15.121:0/176911385 shutdown_connections 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 --2- 172.21.15.121:0/176911385 >> v2:172.21.15.123:3300/0 conn(0x7f3414144530 0x7f3414140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 --2- 172.21.15.121:0/176911385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414143ff0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 -- 172.21.15.121:0/176911385 >> 172.21.15.121:0/176911385 conn(0x7f3414068fd0 msgr2=0x7f3414069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 -- 172.21.15.121:0/176911385 shutdown_connections 2023-09-29T23:07:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.592+0000 7f3419177700 1 -- 172.21.15.121:0/176911385 wait complete. 2023-09-29T23:07:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 Processor -- start 2023-09-29T23:07:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 -- start start 2023-09-29T23:07:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341406a630 0x7f341409f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414150b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3414144530 0x7f3414151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3414151590 con 0x7f34141438a0 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341409eb70 con 0x7f3414144530 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3419177700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34141517a0 con 0x7f341406a630 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f341259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f341259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414150b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53486/0 (socket says 172.21.15.121:53486) 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341259c700 1 -- 172.21.15.121:0/3332215855 learned_addr learned my addr 172.21.15.121:0/3332215855 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.593+0000 7f3412d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341406a630 0x7f341409f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341359e700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3414144530 0x7f3414151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341259c700 1 -- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341406a630 msgr2=0x7f341409f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341259c700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341406a630 0x7f341409f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341259c700 1 -- 172.21.15.121:0/3332215855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3414144530 msgr2=0x7f3414151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341259c700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3414144530 0x7f3414151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341259c700 1 -- 172.21.15.121:0/3332215855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34080094f0 con 0x7f34141438a0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341359e700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3414144530 0x7f3414151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f341259c700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414150b10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3408003080 tx=0x7f3408016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f33faffd700 1 -- 172.21.15.121:0/3332215855 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408011d70 con 0x7f34141438a0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f33faffd700 1 -- 172.21.15.121:0/3332215855 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408020070 con 0x7f34141438a0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341409a140 con 0x7f34141438a0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f33faffd700 1 -- 172.21.15.121:0/3332215855 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340801b6a0 con 0x7f34141438a0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.594+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3414150450 con 0x7f34141438a0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.595+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3414071660 con 0x7f34141438a0 2023-09-29T23:07:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.596+0000 7f33faffd700 1 -- 172.21.15.121:0/3332215855 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f3408022030 con 0x7f34141438a0 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.596+0000 7f33faffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.596+0000 7f33faffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.596+0000 7f33faffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.596+0000 7f33faffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.596+0000 7f33faffd700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f3400098780 0x7f340009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.597+0000 7f3412d9d700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f3400098780 0x7f340009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.597+0000 7f33faffd700 1 -- 172.21.15.121:0/3332215855 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f34080c23a0 con 0x7f34141438a0 2023-09-29T23:07:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.597+0000 7f3412d9d700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f3400098780 0x7f340009ac30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3404006fd0 tx=0x7f34040090f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.601+0000 7f33faffd700 1 -- 172.21.15.121:0/3332215855 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f340808fae0 con 0x7f34141438a0 2023-09-29T23:07:01.821 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.820+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3414071660 con 0x7f34141438a0 2023-09-29T23:07:01.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.821+0000 7f33faffd700 1 -- 172.21.15.121:0/3332215855 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3408093100 con 0x7f34141438a0 2023-09-29T23:07:01.822 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:01.822 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:01.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 10 mgrc shutdown 2023-09-29T23:07:01.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f3400098780 msgr2=0x7f340009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:01.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f3400098780 0x7f340009ac30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3404006fd0 tx=0x7f34040090f0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 msgr2=0x7f3414150b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:01.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414150b10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3408003080 tx=0x7f3408016000 comp rx=0 tx=0).stop 2023-09-29T23:07:01.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 shutdown_connections 2023-09-29T23:07:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f3400098780 0x7f340009ac30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3414144530 0x7f3414151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341406a630 0x7f341409f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 --2- 172.21.15.121:0/3332215855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34141438a0 0x7f3414150b10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 >> 172.21.15.121:0/3332215855 conn(0x7f3414068fd0 msgr2=0x7f34140b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 shutdown_connections 2023-09-29T23:07:01.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:01.823+0000 7f3419177700 1 -- 172.21.15.121:0/3332215855 wait complete. 2023-09-29T23:07:04.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:04.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.965+0000 7f8e37821700 1 Processor -- start 2023-09-29T23:07:04.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.965+0000 7f8e37821700 1 -- start start 2023-09-29T23:07:04.968 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e37821700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e30141440 0x7f8e300a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:04.968 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e37821700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8e300b3cf0 0x7f8e3014b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:04.968 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e37821700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e30141810 con 0x7f8e3009b770 2023-09-29T23:07:04.968 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e37821700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3009a7f0 con 0x7f8e300b3cf0 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e37821700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300a4df0 con 0x7f8e30141440 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e355bd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e30141440 0x7f8e300a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e35dbe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8e300b3cf0 0x7f8e3014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e34dbc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8e3009b770 0x7f8e300630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45748/0 (socket says 172.21.15.121:45748) 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e34dbc700 1 -- 172.21.15.121:0/1828565990 learned_addr learned my addr 172.21.15.121:0/1828565990 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e35dbe700 1 -- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e30141440 msgr2=0x7f8e300a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e35dbe700 1 --2- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e30141440 0x7f8e300a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e35dbe700 1 -- 172.21.15.121:0/1828565990 >> v1:172.21.15.121:6789/0 conn(0x7f8e3009b770 legacy=0x7f8e300630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e35dbe700 1 -- 172.21.15.121:0/1828565990 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e300a50b0 con 0x7f8e300b3cf0 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e27fff700 1 -- 172.21.15.121:0/1828565990 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40016611 0 0) 0x7f8e30141810 con 0x7f8e3009b770 2023-09-29T23:07:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e355bd700 1 --2- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e30141440 0x7f8e300a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e35dbe700 1 --2- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e300b3cf0 0x7f8e3014b190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009e60 tx=0x7f8e28009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6479e6cd6c594d90 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.966+0000 7f8e27fff700 1 -- 172.21.15.121:0/1828565990 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800e070 con 0x7f8e300b3cf0 2023-09-29T23:07:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e27fff700 1 -- 172.21.15.121:0/1828565990 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2800bb60 con 0x7f8e300b3cf0 2023-09-29T23:07:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e27fff700 1 -- 172.21.15.121:0/1828565990 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e28012430 con 0x7f8e300b3cf0 2023-09-29T23:07:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 -- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e300b3cf0 msgr2=0x7f8e3014b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 --2- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e300b3cf0 0x7f8e3014b190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009e60 tx=0x7f8e28009780 comp rx=0 tx=0).stop 2023-09-29T23:07:04.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 -- 172.21.15.121:0/1828565990 shutdown_connections 2023-09-29T23:07:04.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 --2- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e300b3cf0 0x7f8e3014b190 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:04.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 --2- 172.21.15.121:0/1828565990 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e30141440 0x7f8e300a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:04.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 -- 172.21.15.121:0/1828565990 >> 172.21.15.121:0/1828565990 conn(0x7f8e30069e90 msgr2=0x7f8e300b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:04.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 -- 172.21.15.121:0/1828565990 shutdown_connections 2023-09-29T23:07:04.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.967+0000 7f8e37821700 1 -- 172.21.15.121:0/1828565990 wait complete. 2023-09-29T23:07:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.968+0000 7f8e37821700 1 Processor -- start 2023-09-29T23:07:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.968+0000 7f8e37821700 1 -- start start 2023-09-29T23:07:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e37821700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e3009b770 0x7f8e300a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e37821700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e300b3cf0 0x7f8e3009f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e37821700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e30141440 0x7f8e301534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e37821700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300a2b90 con 0x7f8e30141440 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e37821700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300a1c20 con 0x7f8e300b3cf0 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e37821700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300a0ea0 con 0x7f8e3009b770 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e3009b770 0x7f8e300a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e3009b770 0x7f8e300a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:57742/0 (socket says 172.21.15.121:57742) 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e35dbe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e30141440 0x7f8e301534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 -- 172.21.15.121:0/1015280213 learned_addr learned my addr 172.21.15.121:0/1015280213 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e34dbc700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e300b3cf0 0x7f8e3009f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 -- 172.21.15.121:0/1015280213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e300b3cf0 msgr2=0x7f8e3009f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e300b3cf0 0x7f8e3009f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:04.975 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 -- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e30141440 msgr2=0x7f8e301534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e30141440 0x7f8e301534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 -- 172.21.15.121:0/1015280213 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e200093c0 con 0x7f8e3009b770 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e35dbe700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e30141440 0x7f8e301534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e34dbc700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e300b3cf0 0x7f8e3009f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.969+0000 7f8e355bd700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e3009b770 0x7f8e300a2370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000ec80 tx=0x7f8e20009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.970+0000 7f8e257fa700 1 -- 172.21.15.121:0/1015280213 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e20005a00 con 0x7f8e3009b770 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.970+0000 7f8e257fa700 1 -- 172.21.15.121:0/1015280213 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e20005b60 con 0x7f8e3009b770 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.970+0000 7f8e257fa700 1 -- 172.21.15.121:0/1015280213 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2000c920 con 0x7f8e3009b770 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.970+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e28009a20 con 0x7f8e3009b770 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.970+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e300a1190 con 0x7f8e3009b770 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.970+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3006ab60 con 0x7f8e3009b770 2023-09-29T23:07:04.976 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.971+0000 7f8e257fa700 1 -- 172.21.15.121:0/1015280213 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f8e2000cc90 con 0x7f8e3009b770 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.971+0000 7f8e257fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.972+0000 7f8e257fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.972+0000 7f8e257fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.972+0000 7f8e257fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.972+0000 7f8e257fa700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8e14098780 0x7f8e1409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.972+0000 7f8e34dbc700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8e14098780 0x7f8e1409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.972+0000 7f8e257fa700 1 -- 172.21.15.121:0/1015280213 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f8e200c09c0 con 0x7f8e3009b770 2023-09-29T23:07:04.977 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.973+0000 7f8e34dbc700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8e14098780 0x7f8e1409ac30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e30144680 tx=0x7f8e2c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:04.978 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:04.976+0000 7f8e257fa700 1 -- 172.21.15.121:0/1015280213 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e2008e070 con 0x7f8e3009b770 2023-09-29T23:07:05.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.204+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8e30071660 con 0x7f8e3009b770 2023-09-29T23:07:05.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.205+0000 7f8e257fa700 1 -- 172.21.15.121:0/1015280213 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8e20020100 con 0x7f8e3009b770 2023-09-29T23:07:05.206 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:05.206 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.206+0000 7f8e37821700 10 mgrc shutdown 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.206+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8e14098780 msgr2=0x7f8e1409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.206+0000 7f8e37821700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8e14098780 0x7f8e1409ac30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e30144680 tx=0x7f8e2c009040 comp rx=0 tx=0).stop 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.206+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e3009b770 msgr2=0x7f8e300a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.206+0000 7f8e37821700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e3009b770 0x7f8e300a2370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000ec80 tx=0x7f8e20009120 comp rx=0 tx=0).stop 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 shutdown_connections 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8e14098780 0x7f8e1409ac30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e30141440 0x7f8e301534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:05.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e3009b770 0x7f8e300a2370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:05.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 --2- 172.21.15.121:0/1015280213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e300b3cf0 0x7f8e3009f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:05.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 >> 172.21.15.121:0/1015280213 conn(0x7f8e30069e90 msgr2=0x7f8e300a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:05.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 shutdown_connections 2023-09-29T23:07:05.209 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:05.207+0000 7f8e37821700 1 -- 172.21.15.121:0/1015280213 wait complete. 2023-09-29T23:07:08.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:08.349 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.348+0000 7f2dff3a3700 1 Processor -- start 2023-09-29T23:07:08.349 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.348+0000 7f2dff3a3700 1 -- start start 2023-09-29T23:07:08.351 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dff3a3700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2df809b770 0x7f2df80630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:08.351 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dff3a3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df8141810 con 0x7f2df8141440 2023-09-29T23:07:08.351 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dff3a3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df809a7f0 con 0x7f2df80b3cf0 2023-09-29T23:07:08.351 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dff3a3700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df80a4df0 con 0x7f2df809b770 2023-09-29T23:07:08.351 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfc93e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2df809b770 0x7f2df80630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:08.351 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfc93e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2df809b770 0x7f2df80630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35076/0 (socket says 172.21.15.121:35076) 2023-09-29T23:07:08.351 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfc93e700 1 -- 172.21.15.121:0/3941824788 learned_addr learned my addr 172.21.15.121:0/3941824788 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfd13f700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2df8141440 0x7f2df80a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58064/0 (socket says 172.21.15.121:58064) 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfc93e700 1 -- 172.21.15.121:0/3941824788 >> v1:172.21.15.123:6789/0 conn(0x7f2df80b3cf0 legacy=0x7f2df814b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfc93e700 1 -- 172.21.15.121:0/3941824788 >> v1:172.21.15.121:6789/0 conn(0x7f2df8141440 legacy=0x7f2df80a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfc93e700 1 -- 172.21.15.121:0/3941824788 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df80a50b0 con 0x7f2df809b770 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2deffff700 1 -- 172.21.15.121:0/3941824788 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915791542 0 0) 0x7f2df8141810 con 0x7f2df8141440 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2dfc93e700 1 --2- 172.21.15.121:0/3941824788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df809b770 0x7f2df80630e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2df40099b0 tx=0x7f2df4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e08c74f4c890498 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.349+0000 7f2deffff700 1 -- 172.21.15.121:0/3941824788 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df400b820 con 0x7f2df809b770 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2deffff700 1 -- 172.21.15.121:0/3941824788 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400b980 con 0x7f2df809b770 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2deffff700 1 -- 172.21.15.121:0/3941824788 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df400bd20 con 0x7f2df809b770 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2dff3a3700 1 -- 172.21.15.121:0/3941824788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df809b770 msgr2=0x7f2df80630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/3941824788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df809b770 0x7f2df80630e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2df40099b0 tx=0x7f2df4009250 comp rx=0 tx=0).stop 2023-09-29T23:07:08.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2dff3a3700 1 -- 172.21.15.121:0/3941824788 shutdown_connections 2023-09-29T23:07:08.353 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/3941824788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df809b770 0x7f2df80630e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:08.353 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2dff3a3700 1 -- 172.21.15.121:0/3941824788 >> 172.21.15.121:0/3941824788 conn(0x7f2df8069e90 msgr2=0x7f2df80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:08.353 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2dff3a3700 1 -- 172.21.15.121:0/3941824788 shutdown_connections 2023-09-29T23:07:08.353 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.350+0000 7f2dff3a3700 1 -- 172.21.15.121:0/3941824788 wait complete. 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.351+0000 7f2dff3a3700 1 Processor -- start 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.351+0000 7f2dff3a3700 1 -- start start 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dff3a3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df809b770 0x7f2df8143e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dff3a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df80b3cf0 0x7f2df81424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dff3a3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df8141440 0x7f2df8142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dff3a3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df8142f50 con 0x7f2df80b3cf0 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dff3a3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df81443c0 con 0x7f2df8141440 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dff3a3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df8150910 con 0x7f2df809b770 2023-09-29T23:07:08.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df80b3cf0 0x7f2df81424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfd13f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df809b770 0x7f2df8143e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df80b3cf0 0x7f2df81424d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35092/0 (socket says 172.21.15.121:35092) 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 -- 172.21.15.121:0/1945286650 learned_addr learned my addr 172.21.15.121:0/1945286650 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfd940700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df8141440 0x7f2df8142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 -- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df809b770 msgr2=0x7f2df8143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df809b770 0x7f2df8143e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 -- 172.21.15.121:0/1945286650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df8141440 msgr2=0x7f2df8142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df8141440 0x7f2df8142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 -- 172.21.15.121:0/1945286650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df40094f0 con 0x7f2df80b3cf0 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfd940700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df8141440 0x7f2df8142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.352+0000 7f2dfc93e700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df80b3cf0 0x7f2df81424d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2df40099b0 tx=0x7f2df4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:08.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.353+0000 7f2ded7fa700 1 -- 172.21.15.121:0/1945286650 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df401c450 con 0x7f2df80b3cf0 2023-09-29T23:07:08.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.353+0000 7f2ded7fa700 1 -- 172.21.15.121:0/1945286650 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df401c5b0 con 0x7f2df80b3cf0 2023-09-29T23:07:08.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.353+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df81434c0 con 0x7f2df80b3cf0 2023-09-29T23:07:08.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.353+0000 7f2ded7fa700 1 -- 172.21.15.121:0/1945286650 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df401c970 con 0x7f2df80b3cf0 2023-09-29T23:07:08.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.353+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df8143160 con 0x7f2df80b3cf0 2023-09-29T23:07:08.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.354+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df8144590 con 0x7f2df80b3cf0 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.354+0000 7f2ded7fa700 1 -- 172.21.15.121:0/1945286650 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2df401cd20 con 0x7f2df80b3cf0 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.354+0000 7f2ded7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.355+0000 7f2ded7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.355+0000 7f2ded7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.355+0000 7f2ded7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.355+0000 7f2ded7fa700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ddc0987b0 0x7f2ddc09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.355+0000 7f2dfd13f700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ddc0987b0 0x7f2ddc09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:08.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.355+0000 7f2ded7fa700 1 -- 172.21.15.121:0/1945286650 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f2df40c2000 con 0x7f2df80b3cf0 2023-09-29T23:07:08.359 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.355+0000 7f2dfd13f700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ddc0987b0 0x7f2ddc09ac60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2de80012a0 tx=0x7f2de8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:08.360 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.359+0000 7f2ded7fa700 1 -- 172.21.15.121:0/1945286650 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2df8144590 con 0x7f2df80b3cf0 2023-09-29T23:07:08.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.584+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2df8144590 con 0x7f2df80b3cf0 2023-09-29T23:07:08.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.584+0000 7f2ded7fa700 1 -- 172.21.15.121:0/1945286650 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2df8144590 con 0x7f2df80b3cf0 2023-09-29T23:07:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:08.585 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:08.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 10 mgrc shutdown 2023-09-29T23:07:08.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ddc0987b0 msgr2=0x7f2ddc09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:08.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ddc0987b0 0x7f2ddc09ac60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2de80012a0 tx=0x7f2de8009040 comp rx=0 tx=0).stop 2023-09-29T23:07:08.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df80b3cf0 msgr2=0x7f2df81424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:08.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df80b3cf0 0x7f2df81424d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2df40099b0 tx=0x7f2df4016000 comp rx=0 tx=0).stop 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 shutdown_connections 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ddc0987b0 0x7f2ddc09ac60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df8141440 0x7f2df8142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df809b770 0x7f2df8143e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 --2- 172.21.15.121:0/1945286650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df80b3cf0 0x7f2df81424d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 >> 172.21.15.121:0/1945286650 conn(0x7f2df8069e90 msgr2=0x7f2df813f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 shutdown_connections 2023-09-29T23:07:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:08.586+0000 7f2dff3a3700 1 -- 172.21.15.121:0/1945286650 wait complete. 2023-09-29T23:07:11.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24ddf74700 1 Processor -- start 2023-09-29T23:07:11.730 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24ddf74700 1 -- start start 2023-09-29T23:07:11.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24ddf74700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f24d8141440 0x7f24d80a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:11.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24ddf74700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d80b3cf0 0x7f24d814b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:11.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24ddf74700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d8141810 con 0x7f24d809b770 2023-09-29T23:07:11.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24ddf74700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d809a7f0 con 0x7f24d8141440 2023-09-29T23:07:11.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24ddf74700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d80a4df0 con 0x7f24d80b3cf0 2023-09-29T23:07:11.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24d77fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f24d8141440 0x7f24d80a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:11.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24d7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d80b3cf0 0x7f24d814b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24d77fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f24d8141440 0x7f24d80a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35106/0 (socket says 172.21.15.121:35106) 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.729+0000 7f24d7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d80b3cf0 0x7f24d814b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44310/0 (socket says 172.21.15.121:44310) 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d77fe700 1 -- 172.21.15.121:0/3127805112 learned_addr learned my addr 172.21.15.121:0/3127805112 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d7fff700 1 -- 172.21.15.121:0/3127805112 >> v2:172.21.15.121:3300/0 conn(0x7f24d8141440 msgr2=0x7f24d80a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d7fff700 1 --2- 172.21.15.121:0/3127805112 >> v2:172.21.15.121:3300/0 conn(0x7f24d8141440 0x7f24d80a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d7fff700 1 -- 172.21.15.121:0/3127805112 >> v1:172.21.15.123:6789/0 conn(0x7f24d809b770 legacy=0x7f24d80630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d7fff700 1 -- 172.21.15.121:0/3127805112 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80a50b0 con 0x7f24d80b3cf0 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d77fe700 1 --2- 172.21.15.121:0/3127805112 >> v2:172.21.15.121:3300/0 conn(0x7f24d8141440 0x7f24d80a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d7fff700 1 --2- 172.21.15.121:0/3127805112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d80b3cf0 0x7f24d814b190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f24d0009e60 tx=0x7f24d0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95612395634c2e7c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d6ffd700 1 -- 172.21.15.121:0/3127805112 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d000e070 con 0x7f24d80b3cf0 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d6ffd700 1 -- 172.21.15.121:0/3127805112 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d000bb60 con 0x7f24d80b3cf0 2023-09-29T23:07:11.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.730+0000 7f24d6ffd700 1 -- 172.21.15.121:0/3127805112 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0012430 con 0x7f24d80b3cf0 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 -- 172.21.15.121:0/3127805112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d80b3cf0 msgr2=0x7f24d814b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 --2- 172.21.15.121:0/3127805112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d80b3cf0 0x7f24d814b190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f24d0009e60 tx=0x7f24d0009780 comp rx=0 tx=0).stop 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 -- 172.21.15.121:0/3127805112 shutdown_connections 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 --2- 172.21.15.121:0/3127805112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d80b3cf0 0x7f24d814b190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 --2- 172.21.15.121:0/3127805112 >> v2:172.21.15.121:3300/0 conn(0x7f24d8141440 0x7f24d80a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 -- 172.21.15.121:0/3127805112 >> 172.21.15.121:0/3127805112 conn(0x7f24d8069e90 msgr2=0x7f24d80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 -- 172.21.15.121:0/3127805112 shutdown_connections 2023-09-29T23:07:11.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.731+0000 7f24ddf74700 1 -- 172.21.15.121:0/3127805112 wait complete. 2023-09-29T23:07:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 Processor -- start 2023-09-29T23:07:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 -- start start 2023-09-29T23:07:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24d809b770 0x7f24d8143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24d80b3cf0 0x7f24d8142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d8141440 0x7f24d8142680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d8144320 con 0x7f24d80b3cf0 2023-09-29T23:07:11.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d81444d0 con 0x7f24d809b770 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24ddf74700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d8143390 con 0x7f24d8141440 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24d7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d8141440 0x7f24d8142680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24cffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24d80b3cf0 0x7f24d8142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24cffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24d80b3cf0 0x7f24d8142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35112/0 (socket says 172.21.15.121:35112) 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24d7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d8141440 0x7f24d8142680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44316/0 (socket says 172.21.15.121:44316) 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24cffff700 1 -- 172.21.15.121:0/320438970 learned_addr learned my addr 172.21.15.121:0/320438970 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.732+0000 7f24d77fe700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24d809b770 0x7f24d8143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24d7fff700 1 -- 172.21.15.121:0/320438970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24d809b770 msgr2=0x7f24d8143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24d7fff700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24d809b770 0x7f24d8143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24d7fff700 1 -- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24d80b3cf0 msgr2=0x7f24d8142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:11.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24d7fff700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24d80b3cf0 0x7f24d8142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24d7fff700 1 -- 172.21.15.121:0/320438970 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c8009480 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24cffff700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24d80b3cf0 0x7f24d8142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24d7fff700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d8141440 0x7f24d8142680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f24d0016000 tx=0x7f24d0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24cf7fe700 1 -- 172.21.15.121:0/320438970 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0012e60 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24cf7fe700 1 -- 172.21.15.121:0/320438970 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d0007000 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d0009a20 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24cf7fe700 1 -- 172.21.15.121:0/320438970 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0025670 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.733+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d8150620 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.734+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d806ab60 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.735+0000 7f24cf7fe700 1 -- 172.21.15.121:0/320438970 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f24d0003050 con 0x7f24d8141440 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.735+0000 7f24cf7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.736+0000 7f24cf7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:11.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.736+0000 7f24cf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:11.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.736+0000 7f24cf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:11.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.736+0000 7f24cf7fe700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f24b8098780 0x7f24b809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:11.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.736+0000 7f24d77fe700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f24b8098780 0x7f24b809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:11.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.736+0000 7f24cf7fe700 1 -- 172.21.15.121:0/320438970 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f24d00c1f70 con 0x7f24d8141440 2023-09-29T23:07:11.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.736+0000 7f24d77fe700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f24b8098780 0x7f24b809ac30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009e90 tx=0x7f24c8009220 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:11.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.740+0000 7f24cf7fe700 1 -- 172.21.15.121:0/320438970 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24d008f620 con 0x7f24d8141440 2023-09-29T23:07:11.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.959+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f24d8071660 con 0x7f24d8141440 2023-09-29T23:07:11.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.959+0000 7f24cf7fe700 1 -- 172.21.15.121:0/320438970 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f24d0092c40 con 0x7f24d8141440 2023-09-29T23:07:11.960 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:11.961 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 10 mgrc shutdown 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f24b8098780 msgr2=0x7f24b809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f24b8098780 0x7f24b809ac30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009e90 tx=0x7f24c8009220 comp rx=0 tx=0).stop 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d8141440 msgr2=0x7f24d8142680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d8141440 0x7f24d8142680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f24d0016000 tx=0x7f24d0019000 comp rx=0 tx=0).stop 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 shutdown_connections 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f24b8098780 0x7f24b809ac30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f24d8141440 0x7f24d8142680 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24d809b770 0x7f24d8143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 --2- 172.21.15.121:0/320438970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24d80b3cf0 0x7f24d8142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:11.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.961+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 >> 172.21.15.121:0/320438970 conn(0x7f24d8069e90 msgr2=0x7f24d80a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:11.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.962+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 shutdown_connections 2023-09-29T23:07:11.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:11.962+0000 7f24ddf74700 1 -- 172.21.15.121:0/320438970 wait complete. 2023-09-29T23:07:14.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:15.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.104+0000 7f9545e46700 1 Processor -- start 2023-09-29T23:07:15.106 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f9545e46700 1 -- start start 2023-09-29T23:07:15.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f9545e46700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f95400b5020 0x7f954009e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f9545e46700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95400b53f0 con 0x7f954006b470 2023-09-29T23:07:15.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f9545e46700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954006ae80 con 0x7f954013ede0 2023-09-29T23:07:15.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f9545e46700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954013e2e0 con 0x7f95400b5020 2023-09-29T23:07:15.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f953effd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f954006b470 0x7f954013e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58078/0 (socket says 172.21.15.121:58078) 2023-09-29T23:07:15.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f953effd700 1 -- 172.21.15.121:0/1777708866 learned_addr learned my addr 172.21.15.121:0/1777708866 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:07:15.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1777708866 >> v2:172.21.15.123:3300/0 conn(0x7f95400b5020 0x7f954009e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329130486 0 0) 0x7f95400b53f0 con 0x7f954006b470 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.105+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f952c003570 con 0x7f954006b470 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632282290 0 0) 0x7f952c003570 con 0x7f954006b470 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 >> v2:172.21.15.123:3300/0 conn(0x7f95400b5020 msgr2=0x7f954009e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 --2- 172.21.15.121:0/1777708866 >> v2:172.21.15.123:3300/0 conn(0x7f95400b5020 0x7f954009e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 >> v1:172.21.15.123:6789/0 conn(0x7f954013ede0 legacy=0x7f9540148c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954006f390 con 0x7f954006b470 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7f9528001470 con 0x7f954006b470 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 >> v1:172.21.15.121:6789/0 conn(0x7f954006b470 legacy=0x7f954013e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:15.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f952c003a60 0x7f952c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c004810 0x7f952c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f952c0051c0 0x7f952c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c00c0a0 con 0x7f952c0051c0 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c00c210 con 0x7f952c004810 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c00c3d0 con 0x7f952c003a60 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f952c003a60 0x7f952c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953effd700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c004810 0x7f952c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953ffff700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f952c0051c0 0x7f952c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953f7fe700 1 -- 172.21.15.121:0/1777708866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c004810 msgr2=0x7f952c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c004810 0x7f952c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953f7fe700 1 -- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f952c0051c0 msgr2=0x7f952c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f952c0051c0 0x7f952c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953f7fe700 1 -- 172.21.15.121:0/1777708866 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c003570 con 0x7f952c003a60 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953ffff700 1 -- 172.21.15.121:0/1777708866 reap_dead start 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f952c003a60 0x7f952c003e70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9530009c00 tx=0x7f95300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d39914e0826553 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.106+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953000e710 con 0x7f952c003a60 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953000e870 con 0x7f952c003a60 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f953e7fc700 1 -- 172.21.15.121:0/1777708866 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953000ec30 con 0x7f952c003a60 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f9545e46700 1 -- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f952c003a60 msgr2=0x7f952c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f9545e46700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f952c003a60 0x7f952c003e70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9530009c00 tx=0x7f95300091a0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f9545e46700 1 -- 172.21.15.121:0/1777708866 shutdown_connections 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f9545e46700 1 --2- 172.21.15.121:0/1777708866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f952c003a60 0x7f952c003e70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f9545e46700 1 -- 172.21.15.121:0/1777708866 >> 172.21.15.121:0/1777708866 conn(0x7f954006d0b0 msgr2=0x7f954006d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:15.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f9545e46700 1 -- 172.21.15.121:0/1777708866 shutdown_connections 2023-09-29T23:07:15.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.107+0000 7f9545e46700 1 -- 172.21.15.121:0/1777708866 wait complete. 2023-09-29T23:07:15.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.109+0000 7f9545e46700 1 Processor -- start 2023-09-29T23:07:15.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.109+0000 7f9545e46700 1 -- start start 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.109+0000 7f9545e46700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c003a60 0x7f9540096630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.109+0000 7f9545e46700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95400a3f40 0x7f9540139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f9545e46700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f954013f6d0 0x7f9540141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f9545e46700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95400961a0 con 0x7f954013f6d0 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f9545e46700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954014e090 con 0x7f952c003a60 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f9545e46700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954013ffe0 con 0x7f95400a3f40 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f954013f6d0 0x7f9540141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95400a3f40 0x7f9540139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f954013f6d0 0x7f9540141bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35134/0 (socket says 172.21.15.121:35134) 2023-09-29T23:07:15.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 -- 172.21.15.121:0/1899283155 learned_addr learned my addr 172.21.15.121:0/1899283155 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c003a60 0x7f9540096630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 -- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95400a3f40 msgr2=0x7f9540139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95400a3f40 0x7f9540139bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 -- 172.21.15.121:0/1899283155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c003a60 msgr2=0x7f9540096630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c003a60 0x7f9540096630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 -- 172.21.15.121:0/1899283155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95401402a0 con 0x7f954013f6d0 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f953ffff700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f954013f6d0 0x7f9540141bc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f954006e710 tx=0x7f9538001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f95377fe700 1 -- 172.21.15.121:0/1899283155 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9538006e80 con 0x7f954013f6d0 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f95377fe700 1 -- 172.21.15.121:0/1899283155 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9538013420 con 0x7f954013f6d0 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9540096ec0 con 0x7f954013f6d0 2023-09-29T23:07:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f95377fe700 1 -- 172.21.15.121:0/1899283155 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95380137e0 con 0x7f954013f6d0 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.110+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954013fcc0 con 0x7f954013f6d0 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.111+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95400970b0 con 0x7f954013f6d0 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.112+0000 7f95377fe700 1 -- 172.21.15.121:0/1899283155 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9538013b90 con 0x7f954013f6d0 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.112+0000 7f95377fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.112+0000 7f95377fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.112+0000 7f95377fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.112+0000 7f95377fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.113+0000 7f95377fe700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9520098780 0x7f952009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.113+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9520098780 0x7f952009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.113+0000 7f95377fe700 1 -- 172.21.15.121:0/1899283155 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f95380c0140 con 0x7f954013f6d0 2023-09-29T23:07:15.118 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.113+0000 7f953f7fe700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9520098780 0x7f952009ac30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f95300091e0 tx=0x7f953001a000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:15.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.117+0000 7f95377fe700 1 -- 172.21.15.121:0/1899283155 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f953808d7f0 con 0x7f954013f6d0 2023-09-29T23:07:15.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.338+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9540071660 con 0x7f954013f6d0 2023-09-29T23:07:15.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.338+0000 7f95377fe700 1 -- 172.21.15.121:0/1899283155 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9538090e10 con 0x7f954013f6d0 2023-09-29T23:07:15.339 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:15.339 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:15.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 10 mgrc shutdown 2023-09-29T23:07:15.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9520098780 msgr2=0x7f952009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:15.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9520098780 0x7f952009ac30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f95300091e0 tx=0x7f953001a000 comp rx=0 tx=0).stop 2023-09-29T23:07:15.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f954013f6d0 msgr2=0x7f9540141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:15.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f954013f6d0 0x7f9540141bc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f954006e710 tx=0x7f9538001670 comp rx=0 tx=0).stop 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 shutdown_connections 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9520098780 0x7f952009ac30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f954013f6d0 0x7f9540141bc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f952c003a60 0x7f9540096630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 --2- 172.21.15.121:0/1899283155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95400a3f40 0x7f9540139bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 >> 172.21.15.121:0/1899283155 conn(0x7f954006d0b0 msgr2=0x7f954009f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 shutdown_connections 2023-09-29T23:07:15.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:15.340+0000 7f9545e46700 1 -- 172.21.15.121:0/1899283155 wait complete. 2023-09-29T23:07:18.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:18.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.482+0000 7fc2ff419700 1 Processor -- start 2023-09-29T23:07:18.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.482+0000 7fc2ff419700 1 -- start start 2023-09-29T23:07:18.484 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2ff419700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc2f80a6fe0 0x7fc2f806e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2ff419700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f806b290 con 0x7fc2f813e680 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2ff419700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f813f310 con 0x7fc2f806aec0 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2ff419700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f809e880 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2fc9b4700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc2f80a6fe0 0x7fc2f806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2fd9b6700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc2f813e680 0x7fc2f813eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35332/0 (socket says 172.21.15.121:35332) 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2fd9b6700 1 -- 172.21.15.121:0/3030005853 learned_addr learned my addr 172.21.15.121:0/3030005853 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2fc9b4700 1 -- 172.21.15.121:0/3030005853 >> v1:172.21.15.123:6789/0 conn(0x7fc2f806aec0 legacy=0x7fc2f809e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2fc9b4700 1 -- 172.21.15.121:0/3030005853 >> v1:172.21.15.121:6789/0 conn(0x7fc2f813e680 legacy=0x7fc2f813eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2ebfff700 1 -- 172.21.15.121:0/3030005853 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969945872 0 0) 0x7fc2f806b290 con 0x7fc2f813e680 2023-09-29T23:07:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.483+0000 7fc2fc9b4700 1 -- 172.21.15.121:0/3030005853 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f806e110 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2fc9b4700 1 --2- 172.21.15.121:0/3030005853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f806e910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40098e0 tx=0x7fc2f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7f1f2333b0a845 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ebfff700 1 -- 172.21.15.121:0/3030005853 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4009e80 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ebfff700 1 -- 172.21.15.121:0/3030005853 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f400b860 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ebfff700 1 -- 172.21.15.121:0/3030005853 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f400bbe0 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ff419700 1 -- 172.21.15.121:0/3030005853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 msgr2=0x7fc2f806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ff419700 1 --2- 172.21.15.121:0/3030005853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f806e910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40098e0 tx=0x7fc2f40091a0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ff419700 1 -- 172.21.15.121:0/3030005853 shutdown_connections 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ff419700 1 --2- 172.21.15.121:0/3030005853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f806e910 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ff419700 1 -- 172.21.15.121:0/3030005853 >> 172.21.15.121:0/3030005853 conn(0x7fc2f8067440 msgr2=0x7fc2f8067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ff419700 1 -- 172.21.15.121:0/3030005853 shutdown_connections 2023-09-29T23:07:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.484+0000 7fc2ff419700 1 -- 172.21.15.121:0/3030005853 wait complete. 2023-09-29T23:07:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.485+0000 7fc2ff419700 1 Processor -- start 2023-09-29T23:07:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.485+0000 7fc2ff419700 1 -- start start 2023-09-29T23:07:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2ff419700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f806aec0 0x7fc2f8136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2ff419700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f8147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2ff419700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f813e680 0x7fc2f8154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2ff419700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8142480 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2ff419700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8147830 con 0x7fc2f806aec0 2023-09-29T23:07:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2ff419700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8147590 con 0x7fc2f813e680 2023-09-29T23:07:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2fc9b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f8147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2fc9b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f8147050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53342/0 (socket says 172.21.15.121:53342) 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2fc9b4700 1 -- 172.21.15.121:0/1967401366 learned_addr learned my addr 172.21.15.121:0/1967401366 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2fd9b6700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f813e680 0x7fc2f8154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.486+0000 7fc2fd1b5700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f806aec0 0x7fc2f8136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2fc9b4700 1 -- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f813e680 msgr2=0x7fc2f8154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2fc9b4700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f813e680 0x7fc2f8154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2fc9b4700 1 -- 172.21.15.121:0/1967401366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f806aec0 msgr2=0x7fc2f8136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2fc9b4700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f806aec0 0x7fc2f8136bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2fc9b4700 1 -- 172.21.15.121:0/1967401366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f4009420 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2fc9b4700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f8147050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4013000 tx=0x7fc2f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:18.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2e97fa700 1 -- 172.21.15.121:0/1967401366 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f401f070 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2e97fa700 1 -- 172.21.15.121:0/1967401366 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f4000c90 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f8147ad0 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2e97fa700 1 -- 172.21.15.121:0/1967401366 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4025420 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.487+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f813e050 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.488+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f8071660 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.489+0000 7fc2e97fa700 1 -- 172.21.15.121:0/1967401366 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc2f4023070 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.489+0000 7fc2e97fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.489+0000 7fc2e97fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.489+0000 7fc2e97fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.489+0000 7fc2e97fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.489+0000 7fc2e97fa700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc2dc098830 0x7fc2dc09ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:18.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.490+0000 7fc2fd1b5700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc2dc098830 0x7fc2dc09ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:18.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.490+0000 7fc2e97fa700 1 -- 172.21.15.121:0/1967401366 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fc2f40c1780 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.490+0000 7fc2fd1b5700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc2dc098830 0x7fc2dc09ace0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec003860 tx=0x7fc2ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:18.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.494+0000 7fc2e97fa700 1 -- 172.21.15.121:0/1967401366 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc2f408ed90 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.719+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc2f806ee50 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.719+0000 7fc2e97fa700 1 -- 172.21.15.121:0/1967401366 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc2f402a100 con 0x7fc2f80a6fe0 2023-09-29T23:07:18.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:18.720 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:18.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 10 mgrc shutdown 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc2dc098830 msgr2=0x7fc2dc09ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc2dc098830 0x7fc2dc09ace0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec003860 tx=0x7fc2ec009040 comp rx=0 tx=0).stop 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 msgr2=0x7fc2f8147050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f8147050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4013000 tx=0x7fc2f4017000 comp rx=0 tx=0).stop 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 shutdown_connections 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc2dc098830 0x7fc2dc09ace0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.721+0000 7fc2ff419700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2f813e680 0x7fc2f8154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.722+0000 7fc2ff419700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2f806aec0 0x7fc2f8136bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.722+0000 7fc2ff419700 1 --2- 172.21.15.121:0/1967401366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f80a6fe0 0x7fc2f8147050 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:18.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.722+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 >> 172.21.15.121:0/1967401366 conn(0x7fc2f8067440 msgr2=0x7fc2f80b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.722+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 shutdown_connections 2023-09-29T23:07:18.723 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:18.722+0000 7fc2ff419700 1 -- 172.21.15.121:0/1967401366 wait complete. 2023-09-29T23:07:21.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:21.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.863+0000 7f439ebb6700 1 Processor -- start 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.863+0000 7f439ebb6700 1 -- start start 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439ebb6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f439809b770 0x7f43980630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439ebb6700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f43980b3cf0 0x7f439814b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439ebb6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4398141810 con 0x7f4398141440 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439ebb6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439809a7f0 con 0x7f439809b770 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439ebb6700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43980a4df0 con 0x7f43980b3cf0 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f4397fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f439809b770 0x7f43980630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439c952700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4398141440 0x7f43980a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35336/0 (socket says 172.21.15.121:35336) 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f4397fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f439809b770 0x7f43980630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53346/0 (socket says 172.21.15.121:53346) 2023-09-29T23:07:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439c952700 1 -- 172.21.15.121:0/712373337 learned_addr learned my addr 172.21.15.121:0/712373337 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439d153700 1 --2- 172.21.15.121:0/712373337 >> v2:172.21.15.123:3300/0 conn(0x7f43980b3cf0 0x7f439814b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f4397fff700 1 -- 172.21.15.121:0/712373337 >> v2:172.21.15.123:3300/0 conn(0x7f43980b3cf0 msgr2=0x7f439814b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f4397fff700 1 --2- 172.21.15.121:0/712373337 >> v2:172.21.15.123:3300/0 conn(0x7f43980b3cf0 0x7f439814b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f4397fff700 1 -- 172.21.15.121:0/712373337 >> v1:172.21.15.121:6789/0 conn(0x7f4398141440 legacy=0x7f43980a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f4397fff700 1 -- 172.21.15.121:0/712373337 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43980a50b0 con 0x7f439809b770 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f43977fe700 1 -- 172.21.15.121:0/712373337 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53631951 0 0) 0x7f4398141810 con 0x7f4398141440 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.864+0000 7f439d153700 1 --2- 172.21.15.121:0/712373337 >> v2:172.21.15.123:3300/0 conn(0x7f43980b3cf0 0x7f439814b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f4397fff700 1 --2- 172.21.15.121:0/712373337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980630e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f439806b690 tx=0x7f43840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d9aaa15d3f11ff server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f43977fe700 1 -- 172.21.15.121:0/712373337 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438400b6b0 con 0x7f439809b770 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f43977fe700 1 -- 172.21.15.121:0/712373337 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438400b810 con 0x7f439809b770 2023-09-29T23:07:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f43977fe700 1 -- 172.21.15.121:0/712373337 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438400bbb0 con 0x7f439809b770 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 -- 172.21.15.121:0/712373337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 msgr2=0x7f43980630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 --2- 172.21.15.121:0/712373337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980630e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f439806b690 tx=0x7f43840091a0 comp rx=0 tx=0).stop 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 -- 172.21.15.121:0/712373337 shutdown_connections 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 --2- 172.21.15.121:0/712373337 >> v2:172.21.15.123:3300/0 conn(0x7f43980b3cf0 0x7f439814b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 --2- 172.21.15.121:0/712373337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980630e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 -- 172.21.15.121:0/712373337 >> 172.21.15.121:0/712373337 conn(0x7f4398069e90 msgr2=0x7f43980b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 -- 172.21.15.121:0/712373337 shutdown_connections 2023-09-29T23:07:21.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.865+0000 7f439ebb6700 1 -- 172.21.15.121:0/712373337 wait complete. 2023-09-29T23:07:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.866+0000 7f439ebb6700 1 Processor -- start 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.866+0000 7f439ebb6700 1 -- start start 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439ebb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439ebb6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43980b3cf0 0x7f439809de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439ebb6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4398141440 0x7f439813a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439ebb6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4398153910 con 0x7f439809b770 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439ebb6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4398153bd0 con 0x7f43980b3cf0 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439ebb6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4398153e90 con 0x7f4398141440 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:21.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53358/0 (socket says 172.21.15.121:53358) 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 -- 172.21.15.121:0/630080413 learned_addr learned my addr 172.21.15.121:0/630080413 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439d153700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4398141440 0x7f439813a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f4397fff700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43980b3cf0 0x7f439809de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 -- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4398141440 msgr2=0x7f439813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4398141440 0x7f439813a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 -- 172.21.15.121:0/630080413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43980b3cf0 msgr2=0x7f439809de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43980b3cf0 0x7f439809de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.867+0000 7f439c952700 1 -- 172.21.15.121:0/630080413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4384009420 con 0x7f439809b770 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.868+0000 7f439c952700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980b2810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f438c00c5e0 tx=0x7f438c007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:21.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.868+0000 7f4394ff9700 1 -- 172.21.15.121:0/630080413 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c0072e0 con 0x7f439809b770 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.868+0000 7f4394ff9700 1 -- 172.21.15.121:0/630080413 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c007460 con 0x7f439809b770 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.868+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439809ec40 con 0x7f439809b770 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.868+0000 7f4394ff9700 1 -- 172.21.15.121:0/630080413 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c0057f0 con 0x7f439809b770 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.868+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439809f6b0 con 0x7f439809b770 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.869+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43980b34b0 con 0x7f439809b770 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4394ff9700 1 -- 172.21.15.121:0/630080413 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f438c005c00 con 0x7f439809b770 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4394ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4394ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4394ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:21.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4394ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:21.874 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4394ff9700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f437c098780 0x7f437c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:21.874 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4397fff700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f437c098780 0x7f437c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:21.874 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.870+0000 7f4394ff9700 1 -- 172.21.15.121:0/630080413 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f438c0bf900 con 0x7f439809b770 2023-09-29T23:07:21.874 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.871+0000 7f4397fff700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f437c098780 0x7f437c09ac30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f439806b690 tx=0x7f4384017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:21.874 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:21.874+0000 7f4394ff9700 1 -- 172.21.15.121:0/630080413 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f43980b34b0 con 0x7f439809b770 2023-09-29T23:07:22.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.099+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f43980b34b0 con 0x7f439809b770 2023-09-29T23:07:22.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.099+0000 7f4394ff9700 1 -- 172.21.15.121:0/630080413 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f43980b34b0 con 0x7f439809b770 2023-09-29T23:07:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:22.100 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 10 mgrc shutdown 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f437c098780 msgr2=0x7f437c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f437c098780 0x7f437c09ac30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f439806b690 tx=0x7f4384017000 comp rx=0 tx=0).stop 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 msgr2=0x7f43980b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980b2810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f438c00c5e0 tx=0x7f438c007e80 comp rx=0 tx=0).stop 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 shutdown_connections 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f437c098780 0x7f437c09ac30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:22.102 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4398141440 0x7f439813a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:22.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f439809b770 0x7f43980b2810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:22.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 --2- 172.21.15.121:0/630080413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43980b3cf0 0x7f439809de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:22.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 >> 172.21.15.121:0/630080413 conn(0x7f4398069e90 msgr2=0x7f43980a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:22.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 shutdown_connections 2023-09-29T23:07:22.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:22.102+0000 7f439ebb6700 1 -- 172.21.15.121:0/630080413 wait complete. 2023-09-29T23:07:25.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:25.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.244+0000 7f5c4cd64700 1 Processor -- start 2023-09-29T23:07:25.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.244+0000 7f5c4cd64700 1 -- start start 2023-09-29T23:07:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4cd64700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5c48141440 0x7f5c480a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4cd64700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5c4809b770 0x7f5c480630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4cd64700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c48141810 con 0x7f5c480b3cf0 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4cd64700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4809a7f0 con 0x7f5c48141440 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4cd64700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c480a4df0 con 0x7f5c4809b770 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4659c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5c48141440 0x7f5c480a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c46d9d700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5c480b3cf0 0x7f5c48143df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35344/0 (socket says 172.21.15.121:35344) 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c46d9d700 1 -- 172.21.15.121:0/2249742447 learned_addr learned my addr 172.21.15.121:0/2249742447 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c45d9b700 1 --2- 172.21.15.121:0/2249742447 >> v2:172.21.15.123:3300/0 conn(0x7f5c4809b770 0x7f5c480630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4659c700 1 -- 172.21.15.121:0/2249742447 >> v2:172.21.15.123:3300/0 conn(0x7f5c4809b770 msgr2=0x7f5c480630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4659c700 1 --2- 172.21.15.121:0/2249742447 >> v2:172.21.15.123:3300/0 conn(0x7f5c4809b770 0x7f5c480630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4659c700 1 -- 172.21.15.121:0/2249742447 >> v1:172.21.15.121:6789/0 conn(0x7f5c480b3cf0 legacy=0x7f5c48143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4559a700 1 -- 172.21.15.121:0/2249742447 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400556179 0 0) 0x7f5c48141810 con 0x7f5c480b3cf0 2023-09-29T23:07:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4659c700 1 -- 172.21.15.121:0/2249742447 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c480a50b0 con 0x7f5c48141440 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c45d9b700 1 --2- 172.21.15.121:0/2249742447 >> v2:172.21.15.123:3300/0 conn(0x7f5c4809b770 0x7f5c480630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4659c700 1 --2- 172.21.15.121:0/2249742447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c48141440 0x7f5c480a45a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806a4a0 tx=0x7f5c3c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c107e016d3f5a07e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4559a700 1 -- 172.21.15.121:0/2249742447 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c005c10 con 0x7f5c48141440 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4559a700 1 -- 172.21.15.121:0/2249742447 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c005d70 con 0x7f5c48141440 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.245+0000 7f5c4559a700 1 -- 172.21.15.121:0/2249742447 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c006280 con 0x7f5c48141440 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 -- 172.21.15.121:0/2249742447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c48141440 msgr2=0x7f5c480a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/2249742447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c48141440 0x7f5c480a45a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806a4a0 tx=0x7f5c3c009120 comp rx=0 tx=0).stop 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 -- 172.21.15.121:0/2249742447 shutdown_connections 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/2249742447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c48141440 0x7f5c480a45a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/2249742447 >> v2:172.21.15.123:3300/0 conn(0x7f5c4809b770 0x7f5c480630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 -- 172.21.15.121:0/2249742447 >> 172.21.15.121:0/2249742447 conn(0x7f5c48069e90 msgr2=0x7f5c480b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 -- 172.21.15.121:0/2249742447 shutdown_connections 2023-09-29T23:07:25.249 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.246+0000 7f5c4cd64700 1 -- 172.21.15.121:0/2249742447 wait complete. 2023-09-29T23:07:25.251 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.247+0000 7f5c4cd64700 1 Processor -- start 2023-09-29T23:07:25.251 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.247+0000 7f5c4cd64700 1 -- start start 2023-09-29T23:07:25.251 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c4809b770 0x7f5c480b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:25.251 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c480b3cf0 0x7f5c4809e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:25.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c48141440 0x7f5c4813a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:25.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4809d500 con 0x7f5c4809b770 2023-09-29T23:07:25.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480a1440 con 0x7f5c48141440 2023-09-29T23:07:25.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480a1e90 con 0x7f5c480b3cf0 2023-09-29T23:07:25.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c4809b770 0x7f5c480b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:25.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c4809b770 0x7f5c480b2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53374/0 (socket says 172.21.15.121:53374) 2023-09-29T23:07:25.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 -- 172.21.15.121:0/21618336 learned_addr learned my addr 172.21.15.121:0/21618336 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c45d9b700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c480b3cf0 0x7f5c4809e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c46d9d700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c48141440 0x7f5c4813a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 -- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c480b3cf0 msgr2=0x7f5c4809e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c480b3cf0 0x7f5c4809e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 -- 172.21.15.121:0/21618336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c48141440 msgr2=0x7f5c4813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c48141440 0x7f5c4813a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 -- 172.21.15.121:0/21618336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c0093c0 con 0x7f5c4809b770 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4659c700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c4809b770 0x7f5c480b2a50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806ad60 tx=0x7f5c3c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c367fc700 1 -- 172.21.15.121:0/21618336 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00bb10 con 0x7f5c4809b770 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c367fc700 1 -- 172.21.15.121:0/21618336 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c00bc70 con 0x7f5c4809b770 2023-09-29T23:07:25.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c480a2150 con 0x7f5c4809b770 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c367fc700 1 -- 172.21.15.121:0/21618336 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c011420 con 0x7f5c4809b770 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.248+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4809d790 con 0x7f5c4809b770 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.249+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4809dee0 con 0x7f5c4809b770 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.250+0000 7f5c367fc700 1 -- 172.21.15.121:0/21618336 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5c3c0117d0 con 0x7f5c4809b770 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.250+0000 7f5c367fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.251+0000 7f5c367fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.251+0000 7f5c367fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.251+0000 7f5c367fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.251+0000 7f5c367fc700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5c28098780 0x7f5c2809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.251+0000 7f5c45d9b700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5c28098780 0x7f5c2809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.251+0000 7f5c367fc700 1 -- 172.21.15.121:0/21618336 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f5c3c0c21b0 con 0x7f5c4809b770 2023-09-29T23:07:25.254 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.251+0000 7f5c45d9b700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5c28098780 0x7f5c2809ac30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c4813cfd0 tx=0x7f5c38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:25.255 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.255+0000 7f5c367fc700 1 -- 172.21.15.121:0/21618336 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c4809dee0 con 0x7f5c4809b770 2023-09-29T23:07:25.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.477+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5c4809dee0 con 0x7f5c4809b770 2023-09-29T23:07:25.477 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.477+0000 7f5c367fc700 1 -- 172.21.15.121:0/21618336 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5c4809dee0 con 0x7f5c4809b770 2023-09-29T23:07:25.477 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:25.478 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 10 mgrc shutdown 2023-09-29T23:07:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5c28098780 msgr2=0x7f5c2809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5c28098780 0x7f5c2809ac30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c4813cfd0 tx=0x7f5c38008040 comp rx=0 tx=0).stop 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c4809b770 msgr2=0x7f5c480b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c4809b770 0x7f5c480b2a50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806ad60 tx=0x7f5c3c016000 comp rx=0 tx=0).stop 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 shutdown_connections 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5c28098780 0x7f5c2809ac30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c48141440 0x7f5c4813a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c4809b770 0x7f5c480b2a50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 --2- 172.21.15.121:0/21618336 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c480b3cf0 0x7f5c4809e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 >> 172.21.15.121:0/21618336 conn(0x7f5c48069e90 msgr2=0x7f5c4813d240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 shutdown_connections 2023-09-29T23:07:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:25.479+0000 7f5c4cd64700 1 -- 172.21.15.121:0/21618336 wait complete. 2023-09-29T23:07:28.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.623+0000 7f928a28c700 1 Processor -- start 2023-09-29T23:07:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.623+0000 7f928a28c700 1 -- start start 2023-09-29T23:07:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.623+0000 7f928a28c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f92840b3cf0 0x7f9284146900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.623+0000 7f928a28c700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840a0620 con 0x7f92840630e0 2023-09-29T23:07:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.623+0000 7f928a28c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92840a08e0 con 0x7f92840a2180 2023-09-29T23:07:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.623+0000 7f928a28c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928409c8d0 con 0x7f92840b3cf0 2023-09-29T23:07:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9283fff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f92840630e0 0x7f928409c1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:34254/0 (socket says 172.21.15.121:34254) 2023-09-29T23:07:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9283fff700 1 -- 172.21.15.121:0/656419038 learned_addr learned my addr 172.21.15.121:0/656419038 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:07:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f92837fe700 1 --2- 172.21.15.121:0/656419038 >> v2:172.21.15.123:3300/0 conn(0x7f92840b3cf0 0x7f9284146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900682633 0 0) 0x7f92840a0620 con 0x7f92840630e0 2023-09-29T23:07:28.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c003570 con 0x7f92840630e0 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871523776 0 0) 0x7f926c003570 con 0x7f92840630e0 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 >> v2:172.21.15.123:3300/0 conn(0x7f92840b3cf0 msgr2=0x7f9284146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 --2- 172.21.15.121:0/656419038 >> v2:172.21.15.123:3300/0 conn(0x7f92840b3cf0 0x7f9284146900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 >> v1:172.21.15.123:6789/0 conn(0x7f92840a2180 legacy=0x7f9284148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92840a62d0 con 0x7f92840630e0 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7f92780015f0 con 0x7f92840630e0 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 >> v1:172.21.15.121:6789/0 conn(0x7f92840630e0 legacy=0x7f928409c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f926c003a60 0x7f926c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f926c004810 0x7f926c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c0051c0 0x7f926c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928409c8d0 con 0x7f926c004810 2023-09-29T23:07:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c00c0e0 con 0x7f926c003a60 2023-09-29T23:07:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c00c2a0 con 0x7f926c0051c0 2023-09-29T23:07:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f92837fe700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f926c004810 0x7f926c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9283fff700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f926c003a60 0x7f926c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.624+0000 7f9288829700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c0051c0 0x7f926c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f92837fe700 1 -- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c0051c0 msgr2=0x7f926c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:28.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f92837fe700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c0051c0 0x7f926c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f92837fe700 1 -- 172.21.15.121:0/656419038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f926c003a60 msgr2=0x7f926c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f92837fe700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f926c003a60 0x7f926c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f92837fe700 1 -- 172.21.15.121:0/656419038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c003570 con 0x7f926c004810 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f9288829700 1 -- 172.21.15.121:0/656419038 reap_dead start 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f92837fe700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f926c004810 0x7f926c004c80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9274009c00 tx=0x7f92740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2152a7236d6bde98 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400e710 con 0x7f926c004810 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927400e870 con 0x7f926c004810 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f9282ffd700 1 -- 172.21.15.121:0/656419038 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927400ec30 con 0x7f926c004810 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f928a28c700 1 -- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f926c004810 msgr2=0x7f926c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f928a28c700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f926c004810 0x7f926c004c80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9274009c00 tx=0x7f92740091a0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f928a28c700 1 -- 172.21.15.121:0/656419038 shutdown_connections 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f928a28c700 1 --2- 172.21.15.121:0/656419038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f926c004810 0x7f926c004c80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.625+0000 7f928a28c700 1 -- 172.21.15.121:0/656419038 >> 172.21.15.121:0/656419038 conn(0x7f9284069e90 msgr2=0x7f92840b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.626+0000 7f928a28c700 1 -- 172.21.15.121:0/656419038 shutdown_connections 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.626+0000 7f928a28c700 1 -- 172.21.15.121:0/656419038 wait complete. 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.626+0000 7f928a28c700 1 Processor -- start 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f928a28c700 1 -- start start 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f928a28c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c004810 0x7f928409b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f928a28c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f928409d980 0x7f92840a8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f928a28c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92840a6ec0 0x7f9284141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f928a28c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92840a7650 con 0x7f928409d980 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f928a28c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928409e3a0 con 0x7f92840a6ec0 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f928a28c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928409e660 con 0x7f926c004810 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f9283fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c004810 0x7f928409b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f9283fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c004810 0x7f928409b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44298/0 (socket says 172.21.15.121:44298) 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f9283fff700 1 -- 172.21.15.121:0/2462928681 learned_addr learned my addr 172.21.15.121:0/2462928681 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f92837fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f928409d980 0x7f92840a8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.627+0000 7f9288829700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92840a6ec0 0x7f9284141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f9283fff700 1 -- 172.21.15.121:0/2462928681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92840a6ec0 msgr2=0x7f9284141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f9283fff700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92840a6ec0 0x7f9284141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f9283fff700 1 -- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f928409d980 msgr2=0x7f92840a8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f9283fff700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f928409d980 0x7f92840a8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f9283fff700 1 -- 172.21.15.121:0/2462928681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928409cb20 con 0x7f926c004810 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f92837fe700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f928409d980 0x7f92840a8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f9283fff700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c004810 0x7f928409b1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f927800fc00 tx=0x7f9278009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f926bfff700 1 -- 172.21.15.121:0/2462928681 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9278001160 con 0x7f926c004810 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f926bfff700 1 -- 172.21.15.121:0/2462928681 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92780072e0 con 0x7f926c004810 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92840a86c0 con 0x7f926c004810 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f926bfff700 1 -- 172.21.15.121:0/2462928681 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9278007680 con 0x7f926c004810 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.628+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9284142020 con 0x7f926c004810 2023-09-29T23:07:28.640 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.629+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928406ab60 con 0x7f926c004810 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f926bfff700 1 -- 172.21.15.121:0/2462928681 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9278005a00 con 0x7f926c004810 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f926bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f926bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f926bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f926bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f926bfff700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9270098780 0x7f927009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f92837fe700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9270098780 0x7f927009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.630+0000 7f926bfff700 1 -- 172.21.15.121:0/2462928681 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f92780c0590 con 0x7f926c004810 2023-09-29T23:07:28.641 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.631+0000 7f92837fe700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9270098780 0x7f927009ac30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f92740091e0 tx=0x7f927401a000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:28.642 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.634+0000 7f926bfff700 1 -- 172.21.15.121:0/2462928681 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f927808dc40 con 0x7f926c004810 2023-09-29T23:07:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.851+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f92840a7fd0 con 0x7f926c004810 2023-09-29T23:07:28.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.851+0000 7f926bfff700 1 -- 172.21.15.121:0/2462928681 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9278005cb0 con 0x7f926c004810 2023-09-29T23:07:28.851 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:28.851 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 10 mgrc shutdown 2023-09-29T23:07:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9270098780 msgr2=0x7f927009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9270098780 0x7f927009ac30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f92740091e0 tx=0x7f927401a000 comp rx=0 tx=0).stop 2023-09-29T23:07:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c004810 msgr2=0x7f928409b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:28.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c004810 0x7f928409b1f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f927800fc00 tx=0x7f9278009040 comp rx=0 tx=0).stop 2023-09-29T23:07:28.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 shutdown_connections 2023-09-29T23:07:28.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9270098780 0x7f927009ac30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92840a6ec0 0x7f9284141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926c004810 0x7f928409b1f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 --2- 172.21.15.121:0/2462928681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f928409d980 0x7f92840a8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:28.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 >> 172.21.15.121:0/2462928681 conn(0x7f9284069e90 msgr2=0x7f92840b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:28.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 shutdown_connections 2023-09-29T23:07:28.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:28.853+0000 7f928a28c700 1 -- 172.21.15.121:0/2462928681 wait complete. 2023-09-29T23:07:31.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:31.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.997+0000 7f8137157700 1 Processor -- start 2023-09-29T23:07:31.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.997+0000 7f8137157700 1 -- start start 2023-09-29T23:07:31.999 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.997+0000 7f8137157700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f81300a0a70 0x7f813013fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:31.999 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.997+0000 7f8137157700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81300a26f0 con 0x7f81300b7560 2023-09-29T23:07:31.999 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.997+0000 7f8137157700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81300a3650 con 0x7f81301400e0 2023-09-29T23:07:31.999 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.997+0000 7f8137157700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8130140520 con 0x7f81300a0a70 2023-09-29T23:07:31.999 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.997+0000 7f812ffff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f81300a0a70 0x7f813013fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812ffff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f81300a0a70 0x7f813013fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37816/0 (socket says 172.21.15.121:37816) 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812ffff700 1 -- 172.21.15.121:0/4016304277 learned_addr learned my addr 172.21.15.121:0/4016304277 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f8134ef3700 1 --1- 172.21.15.121:0/4016304277 >> v1:172.21.15.121:6789/0 conn(0x7f81300b7560 0x7f81300a1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:34258/0 (socket says 172.21.15.121:34258) 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812ffff700 1 -- 172.21.15.121:0/4016304277 >> v1:172.21.15.123:6789/0 conn(0x7f81301400e0 legacy=0x7f813014aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812ffff700 1 -- 172.21.15.121:0/4016304277 >> v1:172.21.15.121:6789/0 conn(0x7f81300b7560 legacy=0x7f81300a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812ffff700 1 -- 172.21.15.121:0/4016304277 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81300a97a0 con 0x7f81300a0a70 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812f7fe700 1 -- 172.21.15.121:0/4016304277 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153016721 0 0) 0x7f81300a26f0 con 0x7f81300b7560 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812ffff700 1 --2- 172.21.15.121:0/4016304277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81300a0a70 0x7f813013fba0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f81180099b0 tx=0x7f8118009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f7683793db3df9 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812f7fe700 1 -- 172.21.15.121:0/4016304277 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811800b860 con 0x7f81300a0a70 2023-09-29T23:07:32.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812f7fe700 1 -- 172.21.15.121:0/4016304277 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811800b9c0 con 0x7f81300a0a70 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.998+0000 7f812f7fe700 1 -- 172.21.15.121:0/4016304277 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811800bd60 con 0x7f81300a0a70 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.999+0000 7f8137157700 1 -- 172.21.15.121:0/4016304277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81300a0a70 msgr2=0x7f813013fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.999+0000 7f8137157700 1 --2- 172.21.15.121:0/4016304277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81300a0a70 0x7f813013fba0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f81180099b0 tx=0x7f8118009250 comp rx=0 tx=0).stop 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.999+0000 7f8137157700 1 -- 172.21.15.121:0/4016304277 shutdown_connections 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.999+0000 7f8137157700 1 --2- 172.21.15.121:0/4016304277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81300a0a70 0x7f813013fba0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.999+0000 7f8137157700 1 -- 172.21.15.121:0/4016304277 >> 172.21.15.121:0/4016304277 conn(0x7f8130068350 msgr2=0x7f81300b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.999+0000 7f8137157700 1 -- 172.21.15.121:0/4016304277 shutdown_connections 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:31.999+0000 7f8137157700 1 -- 172.21.15.121:0/4016304277 wait complete. 2023-09-29T23:07:32.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 Processor -- start 2023-09-29T23:07:32.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 -- start start 2023-09-29T23:07:32.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81300a0a70 0x7f81300aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:32.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81300b7560 0x7f813013ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:32.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81301400e0 0x7f81300b5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:32.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81300b61d0 con 0x7f81301400e0 2023-09-29T23:07:32.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81300b5f10 con 0x7f81300a0a70 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.000+0000 7f8137157700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81300b68b0 con 0x7f81300b7560 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f812ffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81300b7560 0x7f813013ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f81356f4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81301400e0 0x7f81300b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f812ffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81300b7560 0x7f813013ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44302/0 (socket says 172.21.15.121:44302) 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f812ffff700 1 -- 172.21.15.121:0/3650339429 learned_addr learned my addr 172.21.15.121:0/3650339429 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f8134ef3700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81300a0a70 0x7f81300aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f81356f4700 1 -- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81300b7560 msgr2=0x7f813013ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f81356f4700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81300b7560 0x7f813013ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:32.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f81356f4700 1 -- 172.21.15.121:0/3650339429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81300a0a70 msgr2=0x7f81300aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f81356f4700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81300a0a70 0x7f81300aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f81356f4700 1 -- 172.21.15.121:0/3650339429 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81180094f0 con 0x7f81301400e0 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f81356f4700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81301400e0 0x7f81300b5280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f812800bad0 tx=0x7f8128009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f812cff9700 1 -- 172.21.15.121:0/3650339429 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812800c760 con 0x7f81301400e0 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f812cff9700 1 -- 172.21.15.121:0/3650339429 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812800c8e0 con 0x7f81301400e0 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81300aaca0 con 0x7f81301400e0 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f812cff9700 1 -- 172.21.15.121:0/3650339429 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812800cca0 con 0x7f81301400e0 2023-09-29T23:07:32.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.001+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813013b3d0 con 0x7f81301400e0 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.002+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8130071660 con 0x7f81301400e0 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.003+0000 7f812cff9700 1 -- 172.21.15.121:0/3650339429 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f8128014620 con 0x7f81301400e0 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.003+0000 7f812cff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.004+0000 7f812cff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.004+0000 7f812cff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.004+0000 7f812cff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.004+0000 7f812cff9700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8110098780 0x7f811009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.004+0000 7f8134ef3700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8110098780 0x7f811009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.004+0000 7f812cff9700 1 -- 172.21.15.121:0/3650339429 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f81280c0990 con 0x7f81301400e0 2023-09-29T23:07:32.007 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.004+0000 7f8134ef3700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8110098780 0x7f811009ac30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8120007fd0 tx=0x7f8120007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:32.008 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.007+0000 7f812cff9700 1 -- 172.21.15.121:0/3650339429 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f812808e040 con 0x7f81301400e0 2023-09-29T23:07:32.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.228+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f81300a9a60 con 0x7f81301400e0 2023-09-29T23:07:32.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.229+0000 7f812cff9700 1 -- 172.21.15.121:0/3650339429 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8128022080 con 0x7f81301400e0 2023-09-29T23:07:32.229 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:32.229 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.230+0000 7f8137157700 10 mgrc shutdown 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8110098780 msgr2=0x7f811009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8110098780 0x7f811009ac30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8120007fd0 tx=0x7f8120007d50 comp rx=0 tx=0).stop 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81301400e0 msgr2=0x7f81300b5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81301400e0 0x7f81300b5280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f812800bad0 tx=0x7f8128009760 comp rx=0 tx=0).stop 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 shutdown_connections 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f8110098780 0x7f811009ac30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81301400e0 0x7f81300b5280 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:32.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81300a0a70 0x7f81300aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:32.232 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 --2- 172.21.15.121:0/3650339429 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81300b7560 0x7f813013ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:32.232 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 >> 172.21.15.121:0/3650339429 conn(0x7f8130068350 msgr2=0x7f813013e230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:32.232 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 shutdown_connections 2023-09-29T23:07:32.232 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:32.231+0000 7f8137157700 1 -- 172.21.15.121:0/3650339429 wait complete. 2023-09-29T23:07:35.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:35.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.373+0000 7f2df8485700 1 Processor -- start 2023-09-29T23:07:35.373 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.373+0000 7f2df8485700 1 -- start start 2023-09-29T23:07:35.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df8485700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df00a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:35.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df8485700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2df01458e0 0x7f2df0147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:35.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df8485700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df00a85e0 con 0x7f2df00a6950 2023-09-29T23:07:35.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df8485700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df00a6d20 con 0x7f2df01458e0 2023-09-29T23:07:35.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df8485700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df00a91a0 con 0x7f2df00a73b0 2023-09-29T23:07:35.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df5a20700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df00a8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:35.375 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df6a22700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2df01458e0 0x7f2df0147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df6a22700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2df01458e0 0x7f2df0147cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37834/0 (socket says 172.21.15.121:37834) 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df5a20700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df00a8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44312/0 (socket says 172.21.15.121:44312) 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df6a22700 1 -- 172.21.15.121:0/2410434649 learned_addr learned my addr 172.21.15.121:0/2410434649 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df5a20700 1 -- 172.21.15.121:0/2410434649 >> v2:172.21.15.121:3300/0 conn(0x7f2df01458e0 msgr2=0x7f2df0147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df5a20700 1 --2- 172.21.15.121:0/2410434649 >> v2:172.21.15.121:3300/0 conn(0x7f2df01458e0 0x7f2df0147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df5a20700 1 -- 172.21.15.121:0/2410434649 >> v1:172.21.15.123:6789/0 conn(0x7f2df00a6950 legacy=0x7f2df013d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df5a20700 1 -- 172.21.15.121:0/2410434649 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df00a9440 con 0x7f2df00a73b0 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df6a22700 1 --2- 172.21.15.121:0/2410434649 >> v2:172.21.15.121:3300/0 conn(0x7f2df01458e0 0x7f2df0147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df5a20700 1 --2- 172.21.15.121:0/2410434649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df00a8ae0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2de00098e0 tx=0x7f2de00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfc325e96ad2d799 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df521f700 1 -- 172.21.15.121:0/2410434649 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de000b690 con 0x7f2df00a73b0 2023-09-29T23:07:35.376 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df521f700 1 -- 172.21.15.121:0/2410434649 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de000b7f0 con 0x7f2df00a73b0 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.374+0000 7f2df521f700 1 -- 172.21.15.121:0/2410434649 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de000bb90 con 0x7f2df00a73b0 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 -- 172.21.15.121:0/2410434649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 msgr2=0x7f2df00a8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 --2- 172.21.15.121:0/2410434649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df00a8ae0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2de00098e0 tx=0x7f2de00091a0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 -- 172.21.15.121:0/2410434649 shutdown_connections 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 --2- 172.21.15.121:0/2410434649 >> v2:172.21.15.121:3300/0 conn(0x7f2df01458e0 0x7f2df0147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 --2- 172.21.15.121:0/2410434649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df00a8ae0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 -- 172.21.15.121:0/2410434649 >> 172.21.15.121:0/2410434649 conn(0x7f2df006c010 msgr2=0x7f2df006ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 -- 172.21.15.121:0/2410434649 shutdown_connections 2023-09-29T23:07:35.377 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.375+0000 7f2df8485700 1 -- 172.21.15.121:0/2410434649 wait complete. 2023-09-29T23:07:35.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.376+0000 7f2df8485700 1 Processor -- start 2023-09-29T23:07:35.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.376+0000 7f2df8485700 1 -- start start 2023-09-29T23:07:35.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df00a6950 0x7f2df0136250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:35.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df013c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:35.380 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df01458e0 0x7f2df0151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df0151b70 con 0x7f2df00a6950 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df0151d60 con 0x7f2df01458e0 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df0152020 con 0x7f2df00a73b0 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df6221700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df00a6950 0x7f2df0136250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df5a20700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df013c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df6221700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df00a6950 0x7f2df0136250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37842/0 (socket says 172.21.15.121:37842) 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df6221700 1 -- 172.21.15.121:0/640980012 learned_addr learned my addr 172.21.15.121:0/640980012 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df6a22700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df01458e0 0x7f2df0151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df5a20700 1 -- 172.21.15.121:0/640980012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df01458e0 msgr2=0x7f2df0151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df5a20700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df01458e0 0x7f2df0151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.381 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df5a20700 1 -- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df00a6950 msgr2=0x7f2df0136250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df5a20700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df00a6950 0x7f2df0136250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df5a20700 1 -- 172.21.15.121:0/640980012 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de8009ae0 con 0x7f2df00a73b0 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df6221700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df00a6950 0x7f2df0136250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df6a22700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df01458e0 0x7f2df0151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df5a20700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df013c330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2de0015000 tx=0x7f2de0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2de67fc700 1 -- 172.21.15.121:0/640980012 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0000d10 con 0x7f2df00a73b0 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2de67fc700 1 -- 172.21.15.121:0/640980012 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de001b470 con 0x7f2df00a73b0 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2de67fc700 1 -- 172.21.15.121:0/640980012 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de001b870 con 0x7f2df00a73b0 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de0009420 con 0x7f2df00a73b0 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.377+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df0135d90 con 0x7f2df00a73b0 2023-09-29T23:07:35.382 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.378+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df01360c0 con 0x7f2df00a73b0 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.379+0000 7f2de67fc700 1 -- 172.21.15.121:0/640980012 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2de001bca0 con 0x7f2df00a73b0 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.379+0000 7f2de67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.380+0000 7f2de67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.380+0000 7f2de67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.380+0000 7f2de67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.380+0000 7f2de67fc700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2dd4098780 0x7f2dd409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.380+0000 7f2df6221700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2dd4098780 0x7f2dd409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.380+0000 7f2de67fc700 1 -- 172.21.15.121:0/640980012 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f2de00c2580 con 0x7f2df00a73b0 2023-09-29T23:07:35.383 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.380+0000 7f2df6221700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2dd4098780 0x7f2dd409ac30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2dec007fd0 tx=0x7f2dec007f60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:35.384 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.384+0000 7f2de67fc700 1 -- 172.21.15.121:0/640980012 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2de008fc10 con 0x7f2df00a73b0 2023-09-29T23:07:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.605+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2df0071660 con 0x7f2df00a73b0 2023-09-29T23:07:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.605+0000 7f2de67fc700 1 -- 172.21.15.121:0/640980012 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2de0093230 con 0x7f2df00a73b0 2023-09-29T23:07:35.605 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:35.605 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 10 mgrc shutdown 2023-09-29T23:07:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2dd4098780 msgr2=0x7f2dd409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2dd4098780 0x7f2dd409ac30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2dec007fd0 tx=0x7f2dec007f60 comp rx=0 tx=0).stop 2023-09-29T23:07:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 msgr2=0x7f2df013c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df013c330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2de0015000 tx=0x7f2de0016000 comp rx=0 tx=0).stop 2023-09-29T23:07:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 shutdown_connections 2023-09-29T23:07:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2dd4098780 0x7f2dd409ac30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2df01458e0 0x7f2df0151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2df00a6950 0x7f2df0136250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 --2- 172.21.15.121:0/640980012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2df00a73b0 0x7f2df013c330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 >> 172.21.15.121:0/640980012 conn(0x7f2df006c010 msgr2=0x7f2df00a7840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 shutdown_connections 2023-09-29T23:07:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:35.607+0000 7f2df8485700 1 -- 172.21.15.121:0/640980012 wait complete. 2023-09-29T23:07:38.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:38.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.747+0000 7fca97d64700 1 Processor -- start 2023-09-29T23:07:38.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.748+0000 7fca97d64700 1 -- start start 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.748+0000 7fca97d64700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fca900a5cb0 0x7fca90143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.748+0000 7fca97d64700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca90144100 con 0x7fca900b5190 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.748+0000 7fca97d64700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca900a6b00 con 0x7fca900b5560 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.748+0000 7fca97d64700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca900a6dc0 con 0x7fca900a5cb0 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.748+0000 7fca95b00700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fca900b5190 0x7fca90144590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33852/0 (socket says 172.21.15.121:33852) 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.748+0000 7fca95b00700 1 -- 172.21.15.121:0/984102369 learned_addr learned my addr 172.21.15.121:0/984102369 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833714106 0 0) 0x7fca90144100 con 0x7fca900b5190 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7c003570 con 0x7fca900b5190 2023-09-29T23:07:38.753 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 --2- 172.21.15.121:0/984102369 >> v2:172.21.15.123:3300/0 conn(0x7fca900a5cb0 0x7fca90143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037812205 0 0) 0x7fca7c003570 con 0x7fca900b5190 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 >> v2:172.21.15.123:3300/0 conn(0x7fca900a5cb0 msgr2=0x7fca90143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 --2- 172.21.15.121:0/984102369 >> v2:172.21.15.123:3300/0 conn(0x7fca900a5cb0 0x7fca90143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 >> v1:172.21.15.123:6789/0 conn(0x7fca900b5560 legacy=0x7fca900699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca900a6500 con 0x7fca900b5190 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7fca84001470 con 0x7fca900b5190 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 >> v1:172.21.15.121:6789/0 conn(0x7fca900b5190 legacy=0x7fca90144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca7c003a60 0x7fca7c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c004810 0x7fca7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.754 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca7c0051c0 0x7fca7c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca900a6dc0 con 0x7fca7c0051c0 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c00c160 con 0x7fca7c004810 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c00c320 con 0x7fca7c003a60 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca95b00700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca7c003a60 0x7fca7c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca7c0051c0 0x7fca7c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca952ff700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c004810 0x7fca7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 -- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca7c003a60 msgr2=0x7fca7c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca7c003a60 0x7fca7c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 -- 172.21.15.121:0/984102369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c004810 msgr2=0x7fca7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:38.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c004810 0x7fca7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 -- 172.21.15.121:0/984102369 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c003570 con 0x7fca7c0051c0 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca95b00700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca7c003a60 0x7fca7c003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.749+0000 7fca96301700 1 -- 172.21.15.121:0/984102369 reap_dead start 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca96301700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca7c0051c0 0x7fca7c00ba60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fca8800e210 tx=0x7fca8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b55550f11363ec server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8800eac0 con 0x7fca7c0051c0 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8800f040 con 0x7fca7c0051c0 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca94afe700 1 -- 172.21.15.121:0/984102369 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8800ee70 con 0x7fca7c0051c0 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca97d64700 1 -- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca7c0051c0 msgr2=0x7fca7c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca97d64700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca7c0051c0 0x7fca7c00ba60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fca8800e210 tx=0x7fca8800b040 comp rx=0 tx=0).stop 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca97d64700 1 -- 172.21.15.121:0/984102369 shutdown_connections 2023-09-29T23:07:38.756 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca97d64700 1 --2- 172.21.15.121:0/984102369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca7c0051c0 0x7fca7c00ba60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca97d64700 1 -- 172.21.15.121:0/984102369 >> 172.21.15.121:0/984102369 conn(0x7fca90067f10 msgr2=0x7fca900b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca97d64700 1 -- 172.21.15.121:0/984102369 shutdown_connections 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.750+0000 7fca97d64700 1 -- 172.21.15.121:0/984102369 wait complete. 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.751+0000 7fca97d64700 1 Processor -- start 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.751+0000 7fca97d64700 1 -- start start 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca97d64700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c0051c0 0x7fca900a2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca97d64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca900a0560 0x7fca90141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca97d64700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca90141b60 0x7fca901539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca97d64700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9009b780 con 0x7fca900a0560 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca97d64700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca900a49c0 con 0x7fca7c0051c0 2023-09-29T23:07:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca97d64700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9009b960 con 0x7fca90141b60 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca900a0560 0x7fca90141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca900a0560 0x7fca90141620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53706/0 (socket says 172.21.15.121:53706) 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 -- 172.21.15.121:0/2196158445 learned_addr learned my addr 172.21.15.121:0/2196158445 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca96301700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca90141b60 0x7fca901539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca95b00700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c0051c0 0x7fca900a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 -- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca90141b60 msgr2=0x7fca901539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca90141b60 0x7fca901539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 -- 172.21.15.121:0/2196158445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c0051c0 msgr2=0x7fca900a2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c0051c0 0x7fca900a2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.758 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 -- 172.21.15.121:0/2196158445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca900a3de0 con 0x7fca900a0560 2023-09-29T23:07:38.759 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca952ff700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca900a0560 0x7fca90141620 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fca8c002980 tx=0x7fca8c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:38.759 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca81ffb700 1 -- 172.21.15.121:0/2196158445 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c013600 con 0x7fca900a0560 2023-09-29T23:07:38.759 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca81ffb700 1 -- 172.21.15.121:0/2196158445 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c013760 con 0x7fca900a0560 2023-09-29T23:07:38.759 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca81ffb700 1 -- 172.21.15.121:0/2196158445 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c013b00 con 0x7fca900a0560 2023-09-29T23:07:38.759 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.752+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca900a2ea0 con 0x7fca900a0560 2023-09-29T23:07:38.759 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.753+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca900a4440 con 0x7fca900a0560 2023-09-29T23:07:38.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.754+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca900711d0 con 0x7fca900a0560 2023-09-29T23:07:38.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.754+0000 7fca81ffb700 1 -- 172.21.15.121:0/2196158445 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fca8c01c400 con 0x7fca900a0560 2023-09-29T23:07:38.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.754+0000 7fca81ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:38.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.755+0000 7fca81ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:38.761 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.755+0000 7fca81ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:38.761 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.755+0000 7fca81ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:38.761 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.755+0000 7fca81ffb700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fca74098830 0x7fca7409ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:38.761 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.755+0000 7fca95b00700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fca74098830 0x7fca7409ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:38.761 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.755+0000 7fca81ffb700 1 -- 172.21.15.121:0/2196158445 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fca8c0c0930 con 0x7fca900a0560 2023-09-29T23:07:38.761 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.755+0000 7fca95b00700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fca74098830 0x7fca7409ace0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fca84000fd0 tx=0x7fca84001e50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:38.761 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.759+0000 7fca81ffb700 1 -- 172.21.15.121:0/2196158445 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca8c08df40 con 0x7fca900a0560 2023-09-29T23:07:38.980 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.981+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fca9009c630 con 0x7fca900a0560 2023-09-29T23:07:38.981 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.981+0000 7fca81ffb700 1 -- 172.21.15.121:0/2196158445 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fca8c091560 con 0x7fca900a0560 2023-09-29T23:07:38.981 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:38.981 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:38.983 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 10 mgrc shutdown 2023-09-29T23:07:38.983 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fca74098830 msgr2=0x7fca7409ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:38.983 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fca74098830 0x7fca7409ace0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fca84000fd0 tx=0x7fca84001e50 comp rx=0 tx=0).stop 2023-09-29T23:07:38.983 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca900a0560 msgr2=0x7fca90141620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:38.983 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca900a0560 0x7fca90141620 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fca8c002980 tx=0x7fca8c007d30 comp rx=0 tx=0).stop 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 shutdown_connections 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fca74098830 0x7fca7409ace0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca90141b60 0x7fca901539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7c0051c0 0x7fca900a2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 --2- 172.21.15.121:0/2196158445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca900a0560 0x7fca90141620 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 >> 172.21.15.121:0/2196158445 conn(0x7fca90067f10 msgr2=0x7fca9006e750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 shutdown_connections 2023-09-29T23:07:38.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:38.983+0000 7fca97d64700 1 -- 172.21.15.121:0/2196158445 wait complete. 2023-09-29T23:07:41.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.128+0000 7f29edb1c700 1 Processor -- start 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.128+0000 7f29edb1c700 1 -- start start 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.128+0000 7f29edb1c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e806a630 0x7f29e806da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.128+0000 7f29edb1c700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f29e81438a0 0x7f29e8143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.128+0000 7f29edb1c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e806aa00 con 0x7f29e8144530 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29edb1c700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e81406b0 con 0x7f29e81438a0 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29edb1c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e806df40 con 0x7f29e806a630 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e806a630 0x7f29e806da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e806a630 0x7f29e806da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:38432/0 (socket says 172.21.15.121:38432) 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 -- 172.21.15.121:0/2587073482 learned_addr learned my addr 172.21.15.121:0/2587073482 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e6ffd700 1 --2- 172.21.15.121:0/2587073482 >> v2:172.21.15.123:3300/0 conn(0x7f29e81438a0 0x7f29e8143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 -- 172.21.15.121:0/2587073482 >> v2:172.21.15.123:3300/0 conn(0x7f29e81438a0 msgr2=0x7f29e8143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 --2- 172.21.15.121:0/2587073482 >> v2:172.21.15.123:3300/0 conn(0x7f29e81438a0 0x7f29e8143ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 -- 172.21.15.121:0/2587073482 >> v1:172.21.15.123:6789/0 conn(0x7f29e8144530 legacy=0x7f29e8140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 -- 172.21.15.121:0/2587073482 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e81402b0 con 0x7f29e806a630 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e77fe700 1 --2- 172.21.15.121:0/2587073482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e806a630 0x7f29e806da00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f29d80099b0 tx=0x7f29d8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=314d2139b84cbcfd server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e67fc700 1 -- 172.21.15.121:0/2587073482 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d800b820 con 0x7f29e806a630 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e67fc700 1 -- 172.21.15.121:0/2587073482 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d800b980 con 0x7f29e806a630 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.129+0000 7f29e67fc700 1 -- 172.21.15.121:0/2587073482 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d800bd40 con 0x7f29e806a630 2023-09-29T23:07:42.132 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 -- 172.21.15.121:0/2587073482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e806a630 msgr2=0x7f29e806da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 --2- 172.21.15.121:0/2587073482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e806a630 0x7f29e806da00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f29d80099b0 tx=0x7f29d8009250 comp rx=0 tx=0).stop 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 -- 172.21.15.121:0/2587073482 shutdown_connections 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 --2- 172.21.15.121:0/2587073482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e806a630 0x7f29e806da00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 --2- 172.21.15.121:0/2587073482 >> v2:172.21.15.123:3300/0 conn(0x7f29e81438a0 0x7f29e8143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 -- 172.21.15.121:0/2587073482 >> 172.21.15.121:0/2587073482 conn(0x7f29e8068fd0 msgr2=0x7f29e8069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 -- 172.21.15.121:0/2587073482 shutdown_connections 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.130+0000 7f29edb1c700 1 -- 172.21.15.121:0/2587073482 wait complete. 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.131+0000 7f29edb1c700 1 Processor -- start 2023-09-29T23:07:42.133 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.131+0000 7f29edb1c700 1 -- start start 2023-09-29T23:07:42.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29e806a630 0x7f29e81509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:42.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e81438a0 0x7f29e809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:42.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e8144530 0x7f29e809f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:42.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e809f8f0 con 0x7f29e81438a0 2023-09-29T23:07:42.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e809e990 con 0x7f29e806a630 2023-09-29T23:07:42.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8099e20 con 0x7f29e8144530 2023-09-29T23:07:42.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e81438a0 0x7f29e809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e81438a0 0x7f29e809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53714/0 (socket says 172.21.15.121:53714) 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e8144530 0x7f29e809f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e6ffd700 1 -- 172.21.15.121:0/757596310 learned_addr learned my addr 172.21.15.121:0/757596310 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e77fe700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29e806a630 0x7f29e81509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e7fff700 1 -- 172.21.15.121:0/757596310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29e806a630 msgr2=0x7f29e81509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e7fff700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29e806a630 0x7f29e81509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e7fff700 1 -- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e81438a0 msgr2=0x7f29e809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e7fff700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e81438a0 0x7f29e809ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e7fff700 1 -- 172.21.15.121:0/757596310 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80094f0 con 0x7f29e8144530 2023-09-29T23:07:42.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e6ffd700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e81438a0 0x7f29e809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29e7fff700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e8144530 0x7f29e809f360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29e000bad0 tx=0x7f29e0009760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29cf7fe700 1 -- 172.21.15.121:0/757596310 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e000c740 con 0x7f29e8144530 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29cf7fe700 1 -- 172.21.15.121:0/757596310 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e000c8e0 con 0x7f29e8144530 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e8151390 con 0x7f29e8144530 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29cf7fe700 1 -- 172.21.15.121:0/757596310 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e000cca0 con 0x7f29e8144530 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.132+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e809b860 con 0x7f29e8144530 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.133+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e8150680 con 0x7f29e8144530 2023-09-29T23:07:42.138 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.134+0000 7f29cf7fe700 1 -- 172.21.15.121:0/757596310 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f29e0014620 con 0x7f29e8144530 2023-09-29T23:07:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.134+0000 7f29cf7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.135+0000 7f29cf7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.135+0000 7f29cf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.135+0000 7f29cf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.135+0000 7f29cf7fe700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f29d4098780 0x7f29d409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.135+0000 7f29e77fe700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f29d4098780 0x7f29d409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.135+0000 7f29cf7fe700 1 -- 172.21.15.121:0/757596310 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f29e00c0230 con 0x7f29e8144530 2023-09-29T23:07:42.140 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.135+0000 7f29e77fe700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f29d4098780 0x7f29d409ac30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f29d80099b0 tx=0x7f29d8016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:42.140 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.139+0000 7f29cf7fe700 1 -- 172.21.15.121:0/757596310 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f29e004c5a0 con 0x7f29e8144530 2023-09-29T23:07:42.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.358+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f29e8071660 con 0x7f29e8144530 2023-09-29T23:07:42.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.358+0000 7f29cf7fe700 1 -- 172.21.15.121:0/757596310 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f29e008d8e0 con 0x7f29e8144530 2023-09-29T23:07:42.358 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:42.358 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:42.360 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 10 mgrc shutdown 2023-09-29T23:07:42.360 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f29d4098780 msgr2=0x7f29d409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:42.360 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f29d4098780 0x7f29d409ac30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f29d80099b0 tx=0x7f29d8016000 comp rx=0 tx=0).stop 2023-09-29T23:07:42.360 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e8144530 msgr2=0x7f29e809f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:42.360 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e8144530 0x7f29e809f360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29e000bad0 tx=0x7f29e0009760 comp rx=0 tx=0).stop 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 shutdown_connections 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f29d4098780 0x7f29d409ac30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e8144530 0x7f29e809f360 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29e806a630 0x7f29e81509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 --2- 172.21.15.121:0/757596310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e81438a0 0x7f29e809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 >> 172.21.15.121:0/757596310 conn(0x7f29e8068fd0 msgr2=0x7f29e806ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 shutdown_connections 2023-09-29T23:07:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:42.360+0000 7f29edb1c700 1 -- 172.21.15.121:0/757596310 wait complete. 2023-09-29T23:07:45.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:45.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.501+0000 7f6599926700 1 Processor -- start 2023-09-29T23:07:45.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.501+0000 7f6599926700 1 -- start start 2023-09-29T23:07:45.504 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6599926700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f659406a630 0x7f659406da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:45.504 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6599926700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f65941438a0 0x7f6594143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:45.504 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6599926700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594140970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:45.504 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6599926700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659406aa00 con 0x7f659406a630 2023-09-29T23:07:45.504 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6599926700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65941406b0 con 0x7f65941438a0 2023-09-29T23:07:45.504 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6599926700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659406df40 con 0x7f6594144530 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f659406a630 0x7f659406da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f65937fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f659406a630 0x7f659406da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53720/0 (socket says 172.21.15.121:53720) 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f65937fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594140970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:38442/0 (socket says 172.21.15.121:38442) 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 -- 172.21.15.121:0/1548854050 learned_addr learned my addr 172.21.15.121:0/1548854050 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f65927fc700 1 --2- 172.21.15.121:0/1548854050 >> v2:172.21.15.123:3300/0 conn(0x7f65941438a0 0x7f6594143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 -- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 msgr2=0x7f6594140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 --2- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594140970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 -- 172.21.15.121:0/1548854050 >> v2:172.21.15.123:3300/0 conn(0x7f65941438a0 msgr2=0x7f6594143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 --2- 172.21.15.121:0/1548854050 >> v2:172.21.15.123:3300/0 conn(0x7f65941438a0 0x7f6594143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f6592ffd700 1 -- 172.21.15.121:0/1548854050 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65941402b0 con 0x7f659406a630 2023-09-29T23:07:45.505 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f65937fe700 1 --2- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.502+0000 7f65927fc700 1 --2- 172.21.15.121:0/1548854050 >> v2:172.21.15.123:3300/0 conn(0x7f65941438a0 0x7f6594143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6592ffd700 1 --2- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659406da00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f659406b9c0 tx=0x7f6584009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4569d42438270c34 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6591ffb700 1 -- 172.21.15.121:0/1548854050 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584009e70 con 0x7f659406a630 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6591ffb700 1 -- 172.21.15.121:0/1548854050 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658400b860 con 0x7f659406a630 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6591ffb700 1 -- 172.21.15.121:0/1548854050 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658400bc20 con 0x7f659406a630 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 -- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 msgr2=0x7f659406da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 --2- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659406da00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f659406b9c0 tx=0x7f6584009250 comp rx=0 tx=0).stop 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 -- 172.21.15.121:0/1548854050 shutdown_connections 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 --2- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 --2- 172.21.15.121:0/1548854050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659406da00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.506 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 --2- 172.21.15.121:0/1548854050 >> v2:172.21.15.123:3300/0 conn(0x7f65941438a0 0x7f6594143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.507 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 -- 172.21.15.121:0/1548854050 >> 172.21.15.121:0/1548854050 conn(0x7f6594068fd0 msgr2=0x7f6594069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:45.507 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 -- 172.21.15.121:0/1548854050 shutdown_connections 2023-09-29T23:07:45.507 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.503+0000 7f6599926700 1 -- 172.21.15.121:0/1548854050 wait complete. 2023-09-29T23:07:45.507 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.504+0000 7f6599926700 1 Processor -- start 2023-09-29T23:07:45.507 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6599926700 1 -- start start 2023-09-29T23:07:45.509 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6599926700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659409d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:45.509 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6599926700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65941438a0 0x7f6594153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:45.509 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6599926700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6599926700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659409b470 con 0x7f659406a630 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6599926700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659409e600 con 0x7f65941438a0 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6599926700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659409f110 con 0x7f6594144530 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6592ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659409d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f65937fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6592ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659409d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53734/0 (socket says 172.21.15.121:53734) 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f6592ffd700 1 -- 172.21.15.121:0/4235098599 learned_addr learned my addr 172.21.15.121:0/4235098599 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.505+0000 7f65927fc700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65941438a0 0x7f6594153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6592ffd700 1 -- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 msgr2=0x7f6594154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:45.510 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6592ffd700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6592ffd700 1 -- 172.21.15.121:0/4235098599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65941438a0 msgr2=0x7f6594153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6592ffd700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65941438a0 0x7f6594153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6592ffd700 1 -- 172.21.15.121:0/4235098599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c009a20 con 0x7f659406a630 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f65937fe700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6592ffd700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659409d9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65840094c0 tx=0x7f6584017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f657affd700 1 -- 172.21.15.121:0/4235098599 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584000bc0 con 0x7f659406a630 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f657affd700 1 -- 172.21.15.121:0/4235098599 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658400b9d0 con 0x7f659406a630 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f657affd700 1 -- 172.21.15.121:0/4235098599 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584025420 con 0x7f659406a630 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65840094f0 con 0x7f659406a630 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.506+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659409df90 con 0x7f659406a630 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.507+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6594071660 con 0x7f659406a630 2023-09-29T23:07:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.508+0000 7f657affd700 1 -- 172.21.15.121:0/4235098599 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6584023070 con 0x7f659406a630 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.508+0000 7f657affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.508+0000 7f657affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.508+0000 7f657affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.508+0000 7f657affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.508+0000 7f657affd700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f65800987b0 0x7f658009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.509+0000 7f65927fc700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f65800987b0 0x7f658009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.509+0000 7f657affd700 1 -- 172.21.15.121:0/4235098599 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f65840c1800 con 0x7f659406a630 2023-09-29T23:07:45.512 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.509+0000 7f65927fc700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f65800987b0 0x7f658009ac60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6588006f00 tx=0x7f6588008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:45.513 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.512+0000 7f657affd700 1 -- 172.21.15.121:0/4235098599 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f658408ee10 con 0x7f659406a630 2023-09-29T23:07:45.732 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.732+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f65940b2550 con 0x7f659406a630 2023-09-29T23:07:45.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.733+0000 7f657affd700 1 -- 172.21.15.121:0/4235098599 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6584009300 con 0x7f659406a630 2023-09-29T23:07:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:45.733 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 10 mgrc shutdown 2023-09-29T23:07:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f65800987b0 msgr2=0x7f658009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f65800987b0 0x7f658009ac60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6588006f00 tx=0x7f6588008040 comp rx=0 tx=0).stop 2023-09-29T23:07:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 msgr2=0x7f659409d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659409d9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65840094c0 tx=0x7f6584017000 comp rx=0 tx=0).stop 2023-09-29T23:07:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 shutdown_connections 2023-09-29T23:07:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f65800987b0 0x7f658009ac60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6594144530 0x7f6594154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f659406a630 0x7f659409d9d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 --2- 172.21.15.121:0/4235098599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65941438a0 0x7f6594153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 >> 172.21.15.121:0/4235098599 conn(0x7f6594068fd0 msgr2=0x7f65940a51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 shutdown_connections 2023-09-29T23:07:45.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:45.735+0000 7f6599926700 1 -- 172.21.15.121:0/4235098599 wait complete. 2023-09-29T23:07:48.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:48.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.876+0000 7fc099279700 1 Processor -- start 2023-09-29T23:07:48.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.876+0000 7fc099279700 1 -- start start 2023-09-29T23:07:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc099279700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc09409b770 0x7fc0940630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc099279700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0940b3cf0 0x7fc09414b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc099279700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc094141810 con 0x7fc094141440 2023-09-29T23:07:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc099279700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09409a7f0 con 0x7fc09409b770 2023-09-29T23:07:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc099279700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0940a4df0 con 0x7fc0940b3cf0 2023-09-29T23:07:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0940b3cf0 0x7fc09414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0940b3cf0 0x7fc09414b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:40572/0 (socket says 172.21.15.121:40572) 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 -- 172.21.15.121:0/3588240208 learned_addr learned my addr 172.21.15.121:0/3588240208 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0927fc700 1 --2- 172.21.15.121:0/3588240208 >> v2:172.21.15.123:3300/0 conn(0x7fc09409b770 0x7fc0940630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 -- 172.21.15.121:0/3588240208 >> v2:172.21.15.123:3300/0 conn(0x7fc09409b770 msgr2=0x7fc0940630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 --2- 172.21.15.121:0/3588240208 >> v2:172.21.15.123:3300/0 conn(0x7fc09409b770 0x7fc0940630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 -- 172.21.15.121:0/3588240208 >> v1:172.21.15.123:6789/0 conn(0x7fc094141440 legacy=0x7fc0940a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 -- 172.21.15.121:0/3588240208 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0940a50b0 con 0x7fc0940b3cf0 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.877+0000 7fc0937fe700 1 --2- 172.21.15.121:0/3588240208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0940b3cf0 0x7fc09414b190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009f20 tx=0x7fc08c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=106761b4a4e1ee27 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc091ffb700 1 -- 172.21.15.121:0/3588240208 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c00e070 con 0x7fc0940b3cf0 2023-09-29T23:07:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc091ffb700 1 -- 172.21.15.121:0/3588240208 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c00ba80 con 0x7fc0940b3cf0 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc091ffb700 1 -- 172.21.15.121:0/3588240208 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0092e0 con 0x7fc0940b3cf0 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 -- 172.21.15.121:0/3588240208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0940b3cf0 msgr2=0x7fc09414b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 --2- 172.21.15.121:0/3588240208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0940b3cf0 0x7fc09414b190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009f20 tx=0x7fc08c009840 comp rx=0 tx=0).stop 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 -- 172.21.15.121:0/3588240208 shutdown_connections 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 --2- 172.21.15.121:0/3588240208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0940b3cf0 0x7fc09414b190 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 --2- 172.21.15.121:0/3588240208 >> v2:172.21.15.123:3300/0 conn(0x7fc09409b770 0x7fc0940630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 -- 172.21.15.121:0/3588240208 >> 172.21.15.121:0/3588240208 conn(0x7fc094069e90 msgr2=0x7fc0940b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 -- 172.21.15.121:0/3588240208 shutdown_connections 2023-09-29T23:07:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.878+0000 7fc099279700 1 -- 172.21.15.121:0/3588240208 wait complete. 2023-09-29T23:07:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.880+0000 7fc099279700 1 Processor -- start 2023-09-29T23:07:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.880+0000 7fc099279700 1 -- start start 2023-09-29T23:07:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.880+0000 7fc099279700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc09409b770 0x7fc094144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.880+0000 7fc099279700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0940b3cf0 0x7fc0940a2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.880+0000 7fc099279700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094141440 0x7fc094143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc099279700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0941427d0 con 0x7fc094141440 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc099279700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc094143570 con 0x7fc0940b3cf0 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc099279700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0941429e0 con 0x7fc09409b770 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc09409b770 0x7fc094144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc09409b770 0x7fc094144920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:40582/0 (socket says 172.21.15.121:40582) 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc0937fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094141440 0x7fc094143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 -- 172.21.15.121:0/3134800196 learned_addr learned my addr 172.21.15.121:0/3134800196 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc0927fc700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0940b3cf0 0x7fc0940a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 -- 172.21.15.121:0/3134800196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0940b3cf0 msgr2=0x7fc0940a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0940b3cf0 0x7fc0940a2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 -- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094141440 msgr2=0x7fc094143cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094141440 0x7fc094143cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 -- 172.21.15.121:0/3134800196 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c009ae0 con 0x7fc09409b770 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc0937fe700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094141440 0x7fc094143cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc092ffd700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc09409b770 0x7fc094144920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c540 tx=0x7fc084001070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc07affd700 1 -- 172.21.15.121:0/3134800196 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084011590 con 0x7fc09409b770 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc07affd700 1 -- 172.21.15.121:0/3134800196 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0840116f0 con 0x7fc09409b770 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc094142ca0 con 0x7fc09409b770 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc07affd700 1 -- 172.21.15.121:0/3134800196 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084011ab0 con 0x7fc09409b770 2023-09-29T23:07:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.881+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc094144200 con 0x7fc09409b770 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.882+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09406ab60 con 0x7fc09409b770 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.883+0000 7fc07affd700 1 -- 172.21.15.121:0/3134800196 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc08401b3b0 con 0x7fc09409b770 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.883+0000 7fc07affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.884+0000 7fc07affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.884+0000 7fc07affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.884+0000 7fc07affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.884+0000 7fc07affd700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc080098780 0x7fc08009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.884+0000 7fc0927fc700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc080098780 0x7fc08009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.884+0000 7fc07affd700 1 -- 172.21.15.121:0/3134800196 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fc0840bf890 con 0x7fc09409b770 2023-09-29T23:07:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.884+0000 7fc0927fc700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc080098780 0x7fc08009ac30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0940631c0 tx=0x7fc088008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:48.888+0000 7fc07affd700 1 -- 172.21.15.121:0/3134800196 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc08408ceb0 con 0x7fc09409b770 2023-09-29T23:07:49.108 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.109+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc094071660 con 0x7fc09409b770 2023-09-29T23:07:49.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.109+0000 7fc07affd700 1 -- 172.21.15.121:0/3134800196 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc084017070 con 0x7fc09409b770 2023-09-29T23:07:49.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:49.109 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:49.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 10 mgrc shutdown 2023-09-29T23:07:49.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc080098780 msgr2=0x7fc08009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:49.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc080098780 0x7fc08009ac30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0940631c0 tx=0x7fc088008040 comp rx=0 tx=0).stop 2023-09-29T23:07:49.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc09409b770 msgr2=0x7fc094144920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:49.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc09409b770 0x7fc094144920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c540 tx=0x7fc084001070 comp rx=0 tx=0).stop 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 shutdown_connections 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc080098780 0x7fc08009ac30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc094141440 0x7fc094143cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc09409b770 0x7fc094144920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 --2- 172.21.15.121:0/3134800196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0940b3cf0 0x7fc0940a2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 >> 172.21.15.121:0/3134800196 conn(0x7fc094069e90 msgr2=0x7fc0940a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 shutdown_connections 2023-09-29T23:07:49.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:49.111+0000 7fc099279700 1 -- 172.21.15.121:0/3134800196 wait complete. 2023-09-29T23:07:52.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:52.252 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.252+0000 7fb180f04700 1 Processor -- start 2023-09-29T23:07:52.253 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.252+0000 7fb180f04700 1 -- start start 2023-09-29T23:07:52.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb180f04700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb17c141440 0x7fb17c0a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:52.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb180f04700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c0630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:52.257 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb180f04700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c141810 con 0x7fb17c0b3cf0 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb180f04700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb17c09a7f0 con 0x7fb17c141440 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb180f04700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0a4df0 con 0x7fb17c09b770 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb179d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c0630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb17a59c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb17c141440 0x7fb17c0a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb17a59c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb17c141440 0x7fb17c0a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52458/0 (socket says 172.21.15.121:52458) 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb179d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c0630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:40590/0 (socket says 172.21.15.121:40590) 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb17a59c700 1 -- 172.21.15.121:0/2512842277 learned_addr learned my addr 172.21.15.121:0/2512842277 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb179d9b700 1 -- 172.21.15.121:0/2512842277 >> v2:172.21.15.121:3300/0 conn(0x7fb17c141440 msgr2=0x7fb17c0a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb179d9b700 1 --2- 172.21.15.121:0/2512842277 >> v2:172.21.15.121:3300/0 conn(0x7fb17c141440 0x7fb17c0a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.258 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.253+0000 7fb179d9b700 1 -- 172.21.15.121:0/2512842277 >> v1:172.21.15.123:6789/0 conn(0x7fb17c0b3cf0 legacy=0x7fb17c143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb179d9b700 1 -- 172.21.15.121:0/2512842277 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c0a50b0 con 0x7fb17c09b770 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb17a59c700 1 --2- 172.21.15.121:0/2512842277 >> v2:172.21.15.121:3300/0 conn(0x7fb17c141440 0x7fb17c0a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb179d9b700 1 --2- 172.21.15.121:0/2512842277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c0630e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0098e0 tx=0x7fb16c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=977bcc63a362a7d6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb17959a700 1 -- 172.21.15.121:0/2512842277 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c00b640 con 0x7fb17c09b770 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb17959a700 1 -- 172.21.15.121:0/2512842277 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16c00b7a0 con 0x7fb17c09b770 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb17959a700 1 -- 172.21.15.121:0/2512842277 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c00bb40 con 0x7fb17c09b770 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb180f04700 1 -- 172.21.15.121:0/2512842277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 msgr2=0x7fb17c0630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb180f04700 1 --2- 172.21.15.121:0/2512842277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c0630e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0098e0 tx=0x7fb16c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb180f04700 1 -- 172.21.15.121:0/2512842277 shutdown_connections 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb180f04700 1 --2- 172.21.15.121:0/2512842277 >> v2:172.21.15.121:3300/0 conn(0x7fb17c141440 0x7fb17c0a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb180f04700 1 --2- 172.21.15.121:0/2512842277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c0630e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.254+0000 7fb180f04700 1 -- 172.21.15.121:0/2512842277 >> 172.21.15.121:0/2512842277 conn(0x7fb17c069e90 msgr2=0x7fb17c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.255+0000 7fb180f04700 1 -- 172.21.15.121:0/2512842277 shutdown_connections 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.255+0000 7fb180f04700 1 -- 172.21.15.121:0/2512842277 wait complete. 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.255+0000 7fb180f04700 1 Processor -- start 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb180f04700 1 -- start start 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb180f04700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c06f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb180f04700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb17c0b3cf0 0x7fb17c0a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb180f04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb17c141440 0x7fb17c09ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb180f04700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0a0050 con 0x7fb17c141440 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb180f04700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0a1cf0 con 0x7fb17c0b3cf0 2023-09-29T23:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb180f04700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0a1a10 con 0x7fb17c09b770 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c06f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c06f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:40604/0 (socket says 172.21.15.121:40604) 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb17c141440 0x7fb17c09ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 -- 172.21.15.121:0/3778307272 learned_addr learned my addr 172.21.15.121:0/3778307272 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb179d9b700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb17c0b3cf0 0x7fb17c0a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 -- 172.21.15.121:0/3778307272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb17c0b3cf0 msgr2=0x7fb17c0a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb17c0b3cf0 0x7fb17c0a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 -- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb17c141440 msgr2=0x7fb17c09ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb17c141440 0x7fb17c09ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.256+0000 7fb17a59c700 1 -- 172.21.15.121:0/3778307272 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb170009480 con 0x7fb17c09b770 2023-09-29T23:07:52.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.257+0000 7fb17a59c700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c06f120 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb17000ec80 tx=0x7fb170009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.257+0000 7fb16a7fc700 1 -- 172.21.15.121:0/3778307272 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170005a00 con 0x7fb17c09b770 2023-09-29T23:07:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.257+0000 7fb16a7fc700 1 -- 172.21.15.121:0/3778307272 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170005b60 con 0x7fb17c09b770 2023-09-29T23:07:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.257+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb16c009420 con 0x7fb17c09b770 2023-09-29T23:07:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.257+0000 7fb16a7fc700 1 -- 172.21.15.121:0/3778307272 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17000c850 con 0x7fb17c09b770 2023-09-29T23:07:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.257+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c0a0a40 con 0x7fb17c09b770 2023-09-29T23:07:52.262 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.258+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17c06ab60 con 0x7fb17c09b770 2023-09-29T23:07:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb16a7fc700 1 -- 172.21.15.121:0/3778307272 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb17000cc00 con 0x7fb17c09b770 2023-09-29T23:07:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb16a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb16a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb16a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb16a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb16a7fc700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb15c098780 0x7fb15c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:52.263 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb179d9b700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb15c098780 0x7fb15c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:52.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.259+0000 7fb16a7fc700 1 -- 172.21.15.121:0/3778307272 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fb1700c0db0 con 0x7fb17c09b770 2023-09-29T23:07:52.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.260+0000 7fb179d9b700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb15c098780 0x7fb15c09ac30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0098e0 tx=0x7fb16c017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:52.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.263+0000 7fb16a7fc700 1 -- 172.21.15.121:0/3778307272 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb17008e3d0 con 0x7fb17c09b770 2023-09-29T23:07:52.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.482+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb17c0a0c30 con 0x7fb17c09b770 2023-09-29T23:07:52.482 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.483+0000 7fb16a7fc700 1 -- 172.21.15.121:0/3778307272 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb170020080 con 0x7fb17c09b770 2023-09-29T23:07:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:52.483 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 10 mgrc shutdown 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb15c098780 msgr2=0x7fb15c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb15c098780 0x7fb15c09ac30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0098e0 tx=0x7fb16c017000 comp rx=0 tx=0).stop 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 msgr2=0x7fb17c06f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c06f120 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb17000ec80 tx=0x7fb170009220 comp rx=0 tx=0).stop 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 shutdown_connections 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb15c098780 0x7fb15c09ac30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb17c141440 0x7fb17c09ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb17c09b770 0x7fb17c06f120 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.485 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 --2- 172.21.15.121:0/3778307272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb17c0b3cf0 0x7fb17c0a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:52.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 >> 172.21.15.121:0/3778307272 conn(0x7fb17c069e90 msgr2=0x7fb17c0a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:52.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 shutdown_connections 2023-09-29T23:07:52.486 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:52.485+0000 7fb180f04700 1 -- 172.21.15.121:0/3778307272 wait complete. 2023-09-29T23:07:55.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:55.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.626+0000 7fdd50e2b700 1 Processor -- start 2023-09-29T23:07:55.626 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.626+0000 7fdd50e2b700 1 -- start start 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd50e2b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd4c0b3cf0 0x7fdd4c14c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd50e2b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c141810 con 0x7fdd4c141440 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd50e2b700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c09a7f0 con 0x7fdd4c09b770 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd50e2b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c0a4df0 con 0x7fdd4c0b3cf0 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4ad9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd4c0b3cf0 0x7fdd4c14c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4ad9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd4c0b3cf0 0x7fdd4c14c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52472/0 (socket says 172.21.15.121:52472) 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4a59c700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdd4c141440 0x7fdd4c0a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:41746/0 (socket says 172.21.15.121:41746) 2023-09-29T23:07:55.628 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4ad9d700 1 -- 172.21.15.121:0/4011150424 learned_addr learned my addr 172.21.15.121:0/4011150424 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4ad9d700 1 -- 172.21.15.121:0/4011150424 >> v1:172.21.15.123:6789/0 conn(0x7fdd4c09b770 legacy=0x7fdd4c0630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4ad9d700 1 -- 172.21.15.121:0/4011150424 >> v1:172.21.15.121:6789/0 conn(0x7fdd4c141440 legacy=0x7fdd4c0a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4ad9d700 1 -- 172.21.15.121:0/4011150424 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c0a50b0 con 0x7fdd4c0b3cf0 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4959a700 1 -- 172.21.15.121:0/4011150424 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216062584 0 0) 0x7fdd4c141810 con 0x7fdd4c141440 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4ad9d700 1 --2- 172.21.15.121:0/4011150424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c0b3cf0 0x7fdd4c14c1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009f20 tx=0x7fdd44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9037e2044a9451 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4959a700 1 -- 172.21.15.121:0/4011150424 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400e070 con 0x7fdd4c0b3cf0 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4959a700 1 -- 172.21.15.121:0/4011150424 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4400ba80 con 0x7fdd4c0b3cf0 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.627+0000 7fdd4959a700 1 -- 172.21.15.121:0/4011150424 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd440092e0 con 0x7fdd4c0b3cf0 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.628+0000 7fdd50e2b700 1 -- 172.21.15.121:0/4011150424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c0b3cf0 msgr2=0x7fdd4c14c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.628+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/4011150424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c0b3cf0 0x7fdd4c14c1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009f20 tx=0x7fdd44009840 comp rx=0 tx=0).stop 2023-09-29T23:07:55.629 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.628+0000 7fdd50e2b700 1 -- 172.21.15.121:0/4011150424 shutdown_connections 2023-09-29T23:07:55.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.628+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/4011150424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c0b3cf0 0x7fdd4c14c1a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:55.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.628+0000 7fdd50e2b700 1 -- 172.21.15.121:0/4011150424 >> 172.21.15.121:0/4011150424 conn(0x7fdd4c069e90 msgr2=0x7fdd4c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:55.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.628+0000 7fdd50e2b700 1 -- 172.21.15.121:0/4011150424 shutdown_connections 2023-09-29T23:07:55.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.628+0000 7fdd50e2b700 1 -- 172.21.15.121:0/4011150424 wait complete. 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.629+0000 7fdd50e2b700 1 Processor -- start 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.629+0000 7fdd50e2b700 1 -- start start 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd50e2b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c09b770 0x7fdd4c09f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd50e2b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd4c0b3cf0 0x7fdd4c1437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd50e2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c141440 0x7fdd4c142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd50e2b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0a2330 con 0x7fdd4c141440 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd50e2b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0a25f0 con 0x7fdd4c0b3cf0 2023-09-29T23:07:55.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd50e2b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0a1400 con 0x7fdd4c09b770 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd4a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c09b770 0x7fdd4c09f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd4ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c141440 0x7fdd4c142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd4a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c09b770 0x7fdd4c09f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:40618/0 (socket says 172.21.15.121:40618) 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd4a59c700 1 -- 172.21.15.121:0/354730608 learned_addr learned my addr 172.21.15.121:0/354730608 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.630+0000 7fdd49d9b700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd4c0b3cf0 0x7fdd4c1437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd4ad9d700 1 -- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c09b770 msgr2=0x7fdd4c09f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd4ad9d700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c09b770 0x7fdd4c09f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd4ad9d700 1 -- 172.21.15.121:0/354730608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd4c0b3cf0 msgr2=0x7fdd4c1437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd4ad9d700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd4c0b3cf0 0x7fdd4c1437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd4ad9d700 1 -- 172.21.15.121:0/354730608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd44009ae0 con 0x7fdd4c141440 2023-09-29T23:07:55.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd4ad9d700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c141440 0x7fdd4c142030 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdd44015000 tx=0x7fdd44018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd3a7fc700 1 -- 172.21.15.121:0/354730608 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44007d80 con 0x7fdd4c141440 2023-09-29T23:07:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd3a7fc700 1 -- 172.21.15.121:0/354730608 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd44026460 con 0x7fdd4c141440 2023-09-29T23:07:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4c0a15c0 con 0x7fdd4c141440 2023-09-29T23:07:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd3a7fc700 1 -- 172.21.15.121:0/354730608 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44026820 con 0x7fdd4c141440 2023-09-29T23:07:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.631+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c143eb0 con 0x7fdd4c141440 2023-09-29T23:07:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.632+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c144040 con 0x7fdd4c141440 2023-09-29T23:07:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd3a7fc700 1 -- 172.21.15.121:0/354730608 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fdd44026b90 con 0x7fdd4c141440 2023-09-29T23:07:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd3a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd3a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd3a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd3a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd3a7fc700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd2c098780 0x7fdd2c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd4a59c700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd2c098780 0x7fdd2c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:55.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.633+0000 7fdd3a7fc700 1 -- 172.21.15.121:0/354730608 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fdd440c1eb0 con 0x7fdd4c141440 2023-09-29T23:07:55.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.634+0000 7fdd4a59c700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd2c098780 0x7fdd2c09ac30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c002a30 tx=0x7fdd3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:55.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.637+0000 7fdd3a7fc700 1 -- 172.21.15.121:0/354730608 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd4c144040 con 0x7fdd4c141440 2023-09-29T23:07:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.857+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd4c144040 con 0x7fdd4c141440 2023-09-29T23:07:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.858+0000 7fdd3a7fc700 1 -- 172.21.15.121:0/354730608 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdd4c144040 con 0x7fdd4c141440 2023-09-29T23:07:55.859 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:55.859 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 10 mgrc shutdown 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd2c098780 msgr2=0x7fdd2c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd2c098780 0x7fdd2c09ac30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c002a30 tx=0x7fdd3c009040 comp rx=0 tx=0).stop 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c141440 msgr2=0x7fdd4c142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c141440 0x7fdd4c142030 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdd44015000 tx=0x7fdd44018000 comp rx=0 tx=0).stop 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 shutdown_connections 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd2c098780 0x7fdd2c09ac30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:55.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c141440 0x7fdd4c142030 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:55.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.859+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c09b770 0x7fdd4c09f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:55.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.860+0000 7fdd50e2b700 1 --2- 172.21.15.121:0/354730608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd4c0b3cf0 0x7fdd4c1437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:55.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.860+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 >> 172.21.15.121:0/354730608 conn(0x7fdd4c069e90 msgr2=0x7fdd4c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:55.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.860+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 shutdown_connections 2023-09-29T23:07:55.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:55.860+0000 7fdd50e2b700 1 -- 172.21.15.121:0/354730608 wait complete. 2023-09-29T23:07:58.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:07:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83c5305700 1 Processor -- start 2023-09-29T23:07:59.003 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83c5305700 1 -- start start 2023-09-29T23:07:59.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83c5305700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f83c0140590 0x7f83c00a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:59.004 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83c5305700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c00a1cc0 con 0x7f83c00b7560 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83c5305700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c0140b00 con 0x7f83c00a9150 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83c5305700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c0140dc0 con 0x7f83c0140590 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83be7fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f83c0140590 0x7f83c00a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83be7fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f83c0140590 0x7f83c00a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33558/0 (socket says 172.21.15.121:33558) 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83beffd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f83c00b7560 0x7f83c00a2160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:46278/0 (socket says 172.21.15.121:46278) 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.002+0000 7f83be7fc700 1 -- 172.21.15.121:0/3098353583 learned_addr learned my addr 172.21.15.121:0/3098353583 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83be7fc700 1 -- 172.21.15.121:0/3098353583 >> v1:172.21.15.123:6789/0 conn(0x7f83c00a9150 legacy=0x7f83c00a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83be7fc700 1 -- 172.21.15.121:0/3098353583 >> v1:172.21.15.121:6789/0 conn(0x7f83c00b7560 legacy=0x7f83c00a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83bdffb700 1 -- 172.21.15.121:0/3098353583 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792877698 0 0) 0x7f83c00a1cc0 con 0x7f83c00b7560 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83be7fc700 1 -- 172.21.15.121:0/3098353583 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c00b7930 con 0x7f83c0140590 2023-09-29T23:07:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83be7fc700 1 --2- 172.21.15.121:0/3098353583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c0140590 0x7f83c00a0a70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f83a80099b0 tx=0x7f83a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7315c8736dead server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83bdffb700 1 -- 172.21.15.121:0/3098353583 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800b860 con 0x7f83c0140590 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83bdffb700 1 -- 172.21.15.121:0/3098353583 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a800b9c0 con 0x7f83c0140590 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83bdffb700 1 -- 172.21.15.121:0/3098353583 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800bd80 con 0x7f83c0140590 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83c5305700 1 -- 172.21.15.121:0/3098353583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c0140590 msgr2=0x7f83c00a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.003+0000 7f83c5305700 1 --2- 172.21.15.121:0/3098353583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c0140590 0x7f83c00a0a70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f83a80099b0 tx=0x7f83a8009250 comp rx=0 tx=0).stop 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.004+0000 7f83c5305700 1 -- 172.21.15.121:0/3098353583 shutdown_connections 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.004+0000 7f83c5305700 1 --2- 172.21.15.121:0/3098353583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c0140590 0x7f83c00a0a70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.004+0000 7f83c5305700 1 -- 172.21.15.121:0/3098353583 >> 172.21.15.121:0/3098353583 conn(0x7f83c0068310 msgr2=0x7f83c00b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.004+0000 7f83c5305700 1 -- 172.21.15.121:0/3098353583 shutdown_connections 2023-09-29T23:07:59.006 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.004+0000 7f83c5305700 1 -- 172.21.15.121:0/3098353583 wait complete. 2023-09-29T23:07:59.008 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 Processor -- start 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 -- start start 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c00a9150 0x7f83c0142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83c00b7560 0x7f83c00aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c0140590 0x7f83c013b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c0153da0 con 0x7f83c00a9150 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c0154060 con 0x7f83c0140590 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83c5305700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c00ab350 con 0x7f83c00b7560 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83beffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c00a9150 0x7f83c0142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83beffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c00a9150 0x7f83c0142b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33570/0 (socket says 172.21.15.121:33570) 2023-09-29T23:07:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83beffd700 1 -- 172.21.15.121:0/2769366485 learned_addr learned my addr 172.21.15.121:0/2769366485 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83be7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83c00b7560 0x7f83c00aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.005+0000 7f83bf7fe700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c0140590 0x7f83c013b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83beffd700 1 -- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83c00b7560 msgr2=0x7f83c00aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83beffd700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83c00b7560 0x7f83c00aae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83beffd700 1 -- 172.21.15.121:0/2769366485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c0140590 msgr2=0x7f83c013b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83beffd700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c0140590 0x7f83c013b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83beffd700 1 -- 172.21.15.121:0/2769366485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80094f0 con 0x7f83c00a9150 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83be7fc700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83c00b7560 0x7f83c00aae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83beffd700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c00a9150 0x7f83c0142b50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f83b000c810 tx=0x7f83b0007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83b6ffd700 1 -- 172.21.15.121:0/2769366485 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0005b40 con 0x7f83c00a9150 2023-09-29T23:07:59.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83b6ffd700 1 -- 172.21.15.121:0/2769366485 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b0005ca0 con 0x7f83c00a9150 2023-09-29T23:07:59.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c00b4440 con 0x7f83c00a9150 2023-09-29T23:07:59.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83b6ffd700 1 -- 172.21.15.121:0/2769366485 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0007320 con 0x7f83c00a9150 2023-09-29T23:07:59.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.006+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c0146a10 con 0x7f83c00a9150 2023-09-29T23:07:59.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.007+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c0071660 con 0x7f83c00a9150 2023-09-29T23:07:59.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.008+0000 7f83b6ffd700 1 -- 172.21.15.121:0/2769366485 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f83b001b3b0 con 0x7f83c00a9150 2023-09-29T23:07:59.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.008+0000 7f83b6ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:07:59.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.008+0000 7f83b6ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:07:59.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.008+0000 7f83b6ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:59.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.008+0000 7f83b6ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:07:59.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.008+0000 7f83b6ffd700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f83a0098730 0x7f83a009abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:07:59.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.009+0000 7f83be7fc700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f83a0098730 0x7f83a009abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:07:59.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.009+0000 7f83b6ffd700 1 -- 172.21.15.121:0/2769366485 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f83b00bfed0 con 0x7f83c00a9150 2023-09-29T23:07:59.012 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.009+0000 7f83be7fc700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f83a0098730 0x7f83a009abe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f83a80099b0 tx=0x7f83a8017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:07:59.013 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.012+0000 7f83b6ffd700 1 -- 172.21.15.121:0/2769366485 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83b008d450 con 0x7f83c00a9150 2023-09-29T23:07:59.234 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.233+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f83c00b5c00 con 0x7f83c00a9150 2023-09-29T23:07:59.294 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.234+0000 7f83b6ffd700 1 -- 172.21.15.121:0/2769366485 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f83b0017020 con 0x7f83c00a9150 2023-09-29T23:07:59.295 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:07:59.295 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:07:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 10 mgrc shutdown 2023-09-29T23:07:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f83a0098730 msgr2=0x7f83a009abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f83a0098730 0x7f83a009abe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f83a80099b0 tx=0x7f83a8017000 comp rx=0 tx=0).stop 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c00a9150 msgr2=0x7f83c0142b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c00a9150 0x7f83c0142b50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f83b000c810 tx=0x7f83b0007d90 comp rx=0 tx=0).stop 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 shutdown_connections 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f83a0098730 0x7f83a009abe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c0140590 0x7f83c013b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83c00a9150 0x7f83c0142b50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 --2- 172.21.15.121:0/2769366485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83c00b7560 0x7f83c00aae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 >> 172.21.15.121:0/2769366485 conn(0x7f83c0068310 msgr2=0x7f83c013d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 shutdown_connections 2023-09-29T23:07:59.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:07:59.236+0000 7f83c5305700 1 -- 172.21.15.121:0/2769366485 wait complete. 2023-09-29T23:08:02.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:02.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.427+0000 7f4f6747b700 1 Processor -- start 2023-09-29T23:08:02.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.427+0000 7f4f6747b700 1 -- start start 2023-09-29T23:08:02.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f6747b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4f600b5020 0x7f4f6009e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:02.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f6747b700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4f6013ede0 0x7f4f6006b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:02.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f6747b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f600b53f0 con 0x7f4f6006b470 2023-09-29T23:08:02.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f6747b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6013e2e0 con 0x7f4f600b5020 2023-09-29T23:08:02.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f6747b700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6006ae80 con 0x7f4f6013ede0 2023-09-29T23:08:02.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f65217700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4f600b5020 0x7f4f6009e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f64a16700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4f6006b470 0x7f4f6013e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:46294/0 (socket says 172.21.15.121:46294) 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f65217700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4f600b5020 0x7f4f6009e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33584/0 (socket says 172.21.15.121:33584) 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f64a16700 1 -- 172.21.15.121:0/4176706685 learned_addr learned my addr 172.21.15.121:0/4176706685 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f65a18700 1 --2- 172.21.15.121:0/4176706685 >> v2:172.21.15.123:3300/0 conn(0x7f4f6013ede0 0x7f4f6006b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f65217700 1 -- 172.21.15.121:0/4176706685 >> v2:172.21.15.123:3300/0 conn(0x7f4f6013ede0 msgr2=0x7f4f6006b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f65217700 1 --2- 172.21.15.121:0/4176706685 >> v2:172.21.15.123:3300/0 conn(0x7f4f6013ede0 0x7f4f6006b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f65217700 1 -- 172.21.15.121:0/4176706685 >> v1:172.21.15.121:6789/0 conn(0x7f4f6006b470 legacy=0x7f4f6013e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f65217700 1 -- 172.21.15.121:0/4176706685 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6006f390 con 0x7f4f600b5020 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.428+0000 7f4f57fff700 1 -- 172.21.15.121:0/4176706685 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266824926 0 0) 0x7f4f600b53f0 con 0x7f4f6006b470 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f65217700 1 --2- 172.21.15.121:0/4176706685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f600b5020 0x7f4f6009e260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4f6006a810 tx=0x7f4f500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bac7fe26556409 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f57fff700 1 -- 172.21.15.121:0/4176706685 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5000b6b0 con 0x7f4f600b5020 2023-09-29T23:08:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f57fff700 1 -- 172.21.15.121:0/4176706685 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5000b810 con 0x7f4f600b5020 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f57fff700 1 -- 172.21.15.121:0/4176706685 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5000bbb0 con 0x7f4f600b5020 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 -- 172.21.15.121:0/4176706685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f600b5020 msgr2=0x7f4f6009e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 --2- 172.21.15.121:0/4176706685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f600b5020 0x7f4f6009e260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4f6006a810 tx=0x7f4f500091a0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 -- 172.21.15.121:0/4176706685 shutdown_connections 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 --2- 172.21.15.121:0/4176706685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f600b5020 0x7f4f6009e260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 --2- 172.21.15.121:0/4176706685 >> v2:172.21.15.123:3300/0 conn(0x7f4f6013ede0 0x7f4f6006b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 -- 172.21.15.121:0/4176706685 >> 172.21.15.121:0/4176706685 conn(0x7f4f6006d0b0 msgr2=0x7f4f6006d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 -- 172.21.15.121:0/4176706685 shutdown_connections 2023-09-29T23:08:02.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.429+0000 7f4f6747b700 1 -- 172.21.15.121:0/4176706685 wait complete. 2023-09-29T23:08:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.430+0000 7f4f6747b700 1 Processor -- start 2023-09-29T23:08:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.430+0000 7f4f6747b700 1 -- start start 2023-09-29T23:08:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f6747b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f6006b470 0x7f4f60150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f6747b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f600b5020 0x7f4f60150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f6747b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f6013ede0 0x7f4f6009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f6747b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601512c0 con 0x7f4f6013ede0 2023-09-29T23:08:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f6747b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60150440 con 0x7f4f6006b470 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f6747b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6009b880 con 0x7f4f600b5020 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f6013ede0 0x7f4f6009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f64a16700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f600b5020 0x7f4f60150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f6013ede0 0x7f4f6009a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33590/0 (socket says 172.21.15.121:33590) 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 -- 172.21.15.121:0/213662428 learned_addr learned my addr 172.21.15.121:0/213662428 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65217700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f6006b470 0x7f4f60150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 -- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f600b5020 msgr2=0x7f4f60150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f600b5020 0x7f4f60150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 -- 172.21.15.121:0/213662428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f6006b470 msgr2=0x7f4f60150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f6006b470 0x7f4f60150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f65a18700 1 -- 172.21.15.121:0/213662428 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f50009420 con 0x7f4f6013ede0 2023-09-29T23:08:02.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.431+0000 7f4f64a16700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f600b5020 0x7f4f60150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.432+0000 7f4f65a18700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f6013ede0 0x7f4f6009a9d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4f58007f50 tx=0x7f4f58009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.432+0000 7f4f557fa700 1 -- 172.21.15.121:0/213662428 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800eb70 con 0x7f4f6013ede0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.432+0000 7f4f557fa700 1 -- 172.21.15.121:0/213662428 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5800ecf0 con 0x7f4f6013ede0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.432+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f601514d0 con 0x7f4f6013ede0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.432+0000 7f4f557fa700 1 -- 172.21.15.121:0/213662428 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58014670 con 0x7f4f6013ede0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.432+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f60142290 con 0x7f4f6013ede0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.433+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f60071660 con 0x7f4f6013ede0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.434+0000 7f4f557fa700 1 -- 172.21.15.121:0/213662428 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f4f580149e0 con 0x7f4f6013ede0 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.434+0000 7f4f557fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.434+0000 7f4f557fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.434+0000 7f4f557fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:02.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.434+0000 7f4f557fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:02.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.434+0000 7f4f557fa700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4f440987b0 0x7f4f4409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:02.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.434+0000 7f4f65217700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4f440987b0 0x7f4f4409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:02.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.435+0000 7f4f557fa700 1 -- 172.21.15.121:0/213662428 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f4f580c0a90 con 0x7f4f6013ede0 2023-09-29T23:08:02.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.435+0000 7f4f65217700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4f440987b0 0x7f4f4409ac60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4f500093f0 tx=0x7f4f50017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:02.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.439+0000 7f4f557fa700 1 -- 172.21.15.121:0/213662428 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4f5808e0b0 con 0x7f4f6013ede0 2023-09-29T23:08:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.663+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4f6009e7a0 con 0x7f4f6013ede0 2023-09-29T23:08:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.663+0000 7f4f557fa700 1 -- 172.21.15.121:0/213662428 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4f580916d0 con 0x7f4f6013ede0 2023-09-29T23:08:02.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:02.664 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 10 mgrc shutdown 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4f440987b0 msgr2=0x7f4f4409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4f440987b0 0x7f4f4409ac60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4f500093f0 tx=0x7f4f50017000 comp rx=0 tx=0).stop 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f6013ede0 msgr2=0x7f4f6009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f6013ede0 0x7f4f6009a9d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4f58007f50 tx=0x7f4f58009860 comp rx=0 tx=0).stop 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 shutdown_connections 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4f440987b0 0x7f4f4409ac60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f6013ede0 0x7f4f6009a9d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f6006b470 0x7f4f60150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 --2- 172.21.15.121:0/213662428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f600b5020 0x7f4f60150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 >> 172.21.15.121:0/213662428 conn(0x7f4f6006d0b0 msgr2=0x7f4f60067b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 shutdown_connections 2023-09-29T23:08:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:02.665+0000 7f4f6747b700 1 -- 172.21.15.121:0/213662428 wait complete. 2023-09-29T23:08:05.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:05.807 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.806+0000 7f7cbc932700 1 Processor -- start 2023-09-29T23:08:05.807 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.806+0000 7f7cbc932700 1 -- start start 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cbc932700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb809c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cbc932700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7cb809c700 0x7f7cb80a1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cbc932700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb80a0620 con 0x7f7cb80b3cf0 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cbc932700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb80a08e0 con 0x7f7cb809c700 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cbc932700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb80a62d0 con 0x7f7cb80630e0 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb809c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb5d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7cb809c700 0x7f7cb80a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb5d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7cb809c700 0x7f7cb80a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33600/0 (socket says 172.21.15.121:33600) 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb809c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43316/0 (socket says 172.21.15.121:43316) 2023-09-29T23:08:05.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb5d9b700 1 -- 172.21.15.121:0/3376023124 learned_addr learned my addr 172.21.15.121:0/3376023124 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb5d9b700 1 -- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 msgr2=0x7f7cb809c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb5d9b700 1 --2- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb809c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.807+0000 7f7cb5d9b700 1 -- 172.21.15.121:0/3376023124 >> v1:172.21.15.123:6789/0 conn(0x7f7cb80b3cf0 legacy=0x7f7cb80a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cb5d9b700 1 -- 172.21.15.121:0/3376023124 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80a02c0 con 0x7f7cb809c700 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cb659c700 1 --2- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb809c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cb5d9b700 1 --2- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb809c700 0x7f7cb80a1170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80099b0 tx=0x7f7ca8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61436a7e9cfd3014 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cb559a700 1 -- 172.21.15.121:0/3376023124 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8009ea0 con 0x7f7cb809c700 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cb559a700 1 -- 172.21.15.121:0/3376023124 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca800b860 con 0x7f7cb809c700 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cb559a700 1 -- 172.21.15.121:0/3376023124 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800bc00 con 0x7f7cb809c700 2023-09-29T23:08:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cbc932700 1 -- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb809c700 msgr2=0x7f7cb80a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cbc932700 1 --2- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb809c700 0x7f7cb80a1170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80099b0 tx=0x7f7ca8009250 comp rx=0 tx=0).stop 2023-09-29T23:08:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cbc932700 1 -- 172.21.15.121:0/3376023124 shutdown_connections 2023-09-29T23:08:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cbc932700 1 --2- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb809c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cbc932700 1 --2- 172.21.15.121:0/3376023124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb809c700 0x7f7cb80a1170 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.808+0000 7f7cbc932700 1 -- 172.21.15.121:0/3376023124 >> 172.21.15.121:0/3376023124 conn(0x7f7cb8069e90 msgr2=0x7f7cb80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.809+0000 7f7cbc932700 1 -- 172.21.15.121:0/3376023124 shutdown_connections 2023-09-29T23:08:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.809+0000 7f7cbc932700 1 -- 172.21.15.121:0/3376023124 wait complete. 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.809+0000 7f7cbc932700 1 Processor -- start 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.809+0000 7f7cbc932700 1 -- start start 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cbc932700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb8097960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cbc932700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cb809c700 0x7f7cb8098f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cbc932700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb80b3cf0 0x7f7cb8099ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cbc932700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb806efb0 con 0x7f7cb80b3cf0 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cbc932700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb80982e0 con 0x7f7cb809c700 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cbc932700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8099480 con 0x7f7cb80630e0 2023-09-29T23:08:05.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb8097960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb8097960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43322/0 (socket says 172.21.15.121:43322) 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 -- 172.21.15.121:0/554517770 learned_addr learned my addr 172.21.15.121:0/554517770 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb6d9d700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb80b3cf0 0x7f7cb8099ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb5d9b700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cb809c700 0x7f7cb8098f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 -- 172.21.15.121:0/554517770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cb809c700 msgr2=0x7f7cb8098f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cb809c700 0x7f7cb8098f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 -- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb80b3cf0 msgr2=0x7f7cb8099ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb80b3cf0 0x7f7cb8099ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb659c700 1 -- 172.21.15.121:0/554517770 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0093c0 con 0x7f7cb80630e0 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.810+0000 7f7cb6d9d700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb80b3cf0 0x7f7cb8099ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:05.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.811+0000 7f7cb659c700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb8097960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00ecb0 tx=0x7f7cac009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.811+0000 7f7ca67fc700 1 -- 172.21.15.121:0/554517770 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac005a00 con 0x7f7cb80630e0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.811+0000 7f7ca67fc700 1 -- 172.21.15.121:0/554517770 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac005b60 con 0x7f7cb80630e0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.811+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca80094f0 con 0x7f7cb80630e0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.811+0000 7f7ca67fc700 1 -- 172.21.15.121:0/554517770 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac00c980 con 0x7f7cb80630e0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.811+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb809a1e0 con 0x7f7cb80630e0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.812+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb806ab60 con 0x7f7cb80630e0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7ca67fc700 1 -- 172.21.15.121:0/554517770 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7cac00ccf0 con 0x7f7cb80630e0 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7ca67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7ca67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7ca67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7ca67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:05.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7ca67fc700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7c98098780 0x7f7c9809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:05.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7cb5d9b700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7c98098780 0x7f7c9809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:05.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7ca67fc700 1 -- 172.21.15.121:0/554517770 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f7cac0c09e0 con 0x7f7cb80630e0 2023-09-29T23:08:05.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.813+0000 7f7cb5d9b700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7c98098780 0x7f7c9809ac30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80099b0 tx=0x7f7ca8017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:05.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:05.817+0000 7f7ca67fc700 1 -- 172.21.15.121:0/554517770 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7cac091b20 con 0x7f7cb80630e0 2023-09-29T23:08:06.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.039+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7cb8071660 con 0x7f7cb80630e0 2023-09-29T23:08:06.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.039+0000 7f7ca67fc700 1 -- 172.21.15.121:0/554517770 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7cac020100 con 0x7f7cb80630e0 2023-09-29T23:08:06.040 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:06.040 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:06.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 10 mgrc shutdown 2023-09-29T23:08:06.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7c98098780 msgr2=0x7f7c9809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:06.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7c98098780 0x7f7c9809ac30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80099b0 tx=0x7f7ca8017000 comp rx=0 tx=0).stop 2023-09-29T23:08:06.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 msgr2=0x7f7cb8097960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb8097960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00ecb0 tx=0x7f7cac009120 comp rx=0 tx=0).stop 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 shutdown_connections 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7c98098780 0x7f7c9809ac30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cb80b3cf0 0x7f7cb8099ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cb80630e0 0x7f7cb8097960 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 --2- 172.21.15.121:0/554517770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cb809c700 0x7f7cb8098f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 >> 172.21.15.121:0/554517770 conn(0x7f7cb8069e90 msgr2=0x7f7cb80b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 shutdown_connections 2023-09-29T23:08:06.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:06.041+0000 7f7cbc932700 1 -- 172.21.15.121:0/554517770 wait complete. 2023-09-29T23:08:09.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:09.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.183+0000 7f2fb7494700 1 Processor -- start 2023-09-29T23:08:09.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb7494700 1 -- start start 2023-09-29T23:08:09.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb7494700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb0148b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:09.186 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb7494700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb009dd50 con 0x7f2fb01408c0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb7494700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb009e6d0 con 0x7f2fb00b39c0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb7494700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00a63c0 con 0x7f2fb01413d0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb5a31700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb0148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb4a2f700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2fb01408c0 0x7f2fb0140cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50978/0 (socket says 172.21.15.121:50978) 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.184+0000 7f2fb4a2f700 1 -- 172.21.15.121:0/3172690397 learned_addr learned my addr 172.21.15.121:0/3172690397 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fa7fff700 1 -- 172.21.15.121:0/3172690397 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640181706 0 0) 0x7f2fb009dd50 con 0x7f2fb01408c0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fb5a31700 1 -- 172.21.15.121:0/3172690397 >> v1:172.21.15.123:6789/0 conn(0x7f2fb00b39c0 legacy=0x7f2fb00a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fb5a31700 1 -- 172.21.15.121:0/3172690397 >> v1:172.21.15.121:6789/0 conn(0x7f2fb01408c0 legacy=0x7f2fb0140cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fb5a31700 1 -- 172.21.15.121:0/3172690397 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00a6090 con 0x7f2fb01413d0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fb5a31700 1 --2- 172.21.15.121:0/3172690397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb0148b20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009e90 tx=0x7f2fa8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e3cb589ee72ca9e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fa7fff700 1 -- 172.21.15.121:0/3172690397 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800e070 con 0x7f2fb01413d0 2023-09-29T23:08:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fa7fff700 1 -- 172.21.15.121:0/3172690397 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa800bc90 con 0x7f2fb01413d0 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.185+0000 7f2fa7fff700 1 -- 172.21.15.121:0/3172690397 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8012680 con 0x7f2fb01413d0 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.186+0000 7f2fb7494700 1 -- 172.21.15.121:0/3172690397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 msgr2=0x7f2fb0148b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.186+0000 7f2fb7494700 1 --2- 172.21.15.121:0/3172690397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb0148b20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009e90 tx=0x7f2fa8009780 comp rx=0 tx=0).stop 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.186+0000 7f2fb7494700 1 -- 172.21.15.121:0/3172690397 shutdown_connections 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.186+0000 7f2fb7494700 1 --2- 172.21.15.121:0/3172690397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb0148b20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.186+0000 7f2fb7494700 1 -- 172.21.15.121:0/3172690397 >> 172.21.15.121:0/3172690397 conn(0x7f2fb0069e90 msgr2=0x7f2fb00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.186+0000 7f2fb7494700 1 -- 172.21.15.121:0/3172690397 shutdown_connections 2023-09-29T23:08:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.186+0000 7f2fb7494700 1 -- 172.21.15.121:0/3172690397 wait complete. 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 Processor -- start 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 -- start start 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00b39c0 0x7f2fb00982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb01408c0 0x7f2fb013df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0098800 con 0x7f2fb00b39c0 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb014dec0 con 0x7f2fb01408c0 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb7494700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb014e180 con 0x7f2fb01413d0 2023-09-29T23:08:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.187+0000 7f2fb5230700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00b39c0 0x7f2fb00982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5a31700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00b39c0 0x7f2fb00982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45558/0 (socket says 172.21.15.121:45558) 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 -- 172.21.15.121:0/2301669484 learned_addr learned my addr 172.21.15.121:0/2301669484 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5a31700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb006dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:51510/0 (socket says 172.21.15.121:51510) 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb4a2f700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb01408c0 0x7f2fb013df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 -- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 msgr2=0x7f2fb006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb006dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 -- 172.21.15.121:0/2301669484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb01408c0 msgr2=0x7f2fb013df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb01408c0 0x7f2fb013df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 -- 172.21.15.121:0/2301669484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa8009a20 con 0x7f2fb00b39c0 2023-09-29T23:08:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5a31700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb5230700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00b39c0 0x7f2fb00982c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c790 tx=0x7f2fa0007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fa57fa700 1 -- 172.21.15.121:0/2301669484 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa000e4a0 con 0x7f2fb00b39c0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fa57fa700 1 -- 172.21.15.121:0/2301669484 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa000e620 con 0x7f2fb00b39c0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb0098c40 con 0x7f2fb00b39c0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fa57fa700 1 -- 172.21.15.121:0/2301669484 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa000e9e0 con 0x7f2fb00b39c0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.188+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb00974d0 con 0x7f2fb00b39c0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.189+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb006ab60 con 0x7f2fb00b39c0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.190+0000 7f2fa57fa700 1 -- 172.21.15.121:0/2301669484 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2fa001b3b0 con 0x7f2fb00b39c0 2023-09-29T23:08:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.190+0000 7f2fa57fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.191+0000 7f2fa57fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.191+0000 7f2fa57fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.191+0000 7f2fa57fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.191+0000 7f2fa57fa700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2f94098780 0x7f2f9409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.191+0000 7f2fb4a2f700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2f94098780 0x7f2f9409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.191+0000 7f2fa57fa700 1 -- 172.21.15.121:0/2301669484 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f2fa00bf980 con 0x7f2fb00b39c0 2023-09-29T23:08:09.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.191+0000 7f2fb4a2f700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2f94098780 0x7f2f9409ac30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2fb006e650 tx=0x7f2fac007d10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:09.196 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.195+0000 7f2fa57fa700 1 -- 172.21.15.121:0/2301669484 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2fa008cfa0 con 0x7f2fb00b39c0 2023-09-29T23:08:09.414 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.414+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2fb0071660 con 0x7f2fb00b39c0 2023-09-29T23:08:09.415 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.414+0000 7f2fa57fa700 1 -- 172.21.15.121:0/2301669484 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2fa0017070 con 0x7f2fb00b39c0 2023-09-29T23:08:09.415 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:09.415 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:09.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 10 mgrc shutdown 2023-09-29T23:08:09.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2f94098780 msgr2=0x7f2f9409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:09.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2f94098780 0x7f2f9409ac30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2fb006e650 tx=0x7f2fac007d10 comp rx=0 tx=0).stop 2023-09-29T23:08:09.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00b39c0 msgr2=0x7f2fb00982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:09.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00b39c0 0x7f2fb00982c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c790 tx=0x7f2fa0007f60 comp rx=0 tx=0).stop 2023-09-29T23:08:09.417 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 shutdown_connections 2023-09-29T23:08:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2f94098780 0x7f2f9409ac30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fb01413d0 0x7f2fb006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fb00b39c0 0x7f2fb00982c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 --2- 172.21.15.121:0/2301669484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb01408c0 0x7f2fb013df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 >> 172.21.15.121:0/2301669484 conn(0x7f2fb0069e90 msgr2=0x7f2fb00630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 shutdown_connections 2023-09-29T23:08:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:09.416+0000 7f2fb7494700 1 -- 172.21.15.121:0/2301669484 wait complete. 2023-09-29T23:08:12.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:12.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.557+0000 7fdeed811700 1 Processor -- start 2023-09-29T23:08:12.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.557+0000 7fdeed811700 1 -- start start 2023-09-29T23:08:12.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdeed811700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee80630e0 0x7fdee809c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:12.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdeed811700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdee80b3cf0 0x7fdee80a1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:12.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdeed811700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee80a0620 con 0x7fdee809c700 2023-09-29T23:08:12.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdeed811700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee80a08e0 con 0x7fdee80b3cf0 2023-09-29T23:08:12.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdeed811700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee80a62d0 con 0x7fdee80630e0 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee6ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee80630e0 0x7fdee809c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee77fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdee80b3cf0 0x7fdee80a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee6ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee80630e0 0x7fdee809c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:51522/0 (socket says 172.21.15.121:51522) 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee6ffd700 1 -- 172.21.15.121:0/2918564056 learned_addr learned my addr 172.21.15.121:0/2918564056 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee77fe700 1 -- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee80630e0 msgr2=0x7fdee809c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee77fe700 1 --2- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee80630e0 0x7fdee809c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee77fe700 1 -- 172.21.15.121:0/2918564056 >> v1:172.21.15.123:6789/0 conn(0x7fdee809c700 legacy=0x7fdee80a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee77fe700 1 -- 172.21.15.121:0/2918564056 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee80a02c0 con 0x7fdee80b3cf0 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee6ffd700 1 --2- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee80630e0 0x7fdee809c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee77fe700 1 --2- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a1880 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009e60 tx=0x7fdee0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154d75314b5c8c50 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee5ffb700 1 -- 172.21.15.121:0/2918564056 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee000e070 con 0x7fdee80b3cf0 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.558+0000 7fdee5ffb700 1 -- 172.21.15.121:0/2918564056 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee000bb60 con 0x7fdee80b3cf0 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdee5ffb700 1 -- 172.21.15.121:0/2918564056 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0012430 con 0x7fdee80b3cf0 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 -- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80b3cf0 msgr2=0x7fdee80a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 --2- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a1880 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009e60 tx=0x7fdee0009780 comp rx=0 tx=0).stop 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 -- 172.21.15.121:0/2918564056 shutdown_connections 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 --2- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a1880 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 --2- 172.21.15.121:0/2918564056 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee80630e0 0x7fdee809c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 -- 172.21.15.121:0/2918564056 >> 172.21.15.121:0/2918564056 conn(0x7fdee8069e90 msgr2=0x7fdee80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 -- 172.21.15.121:0/2918564056 shutdown_connections 2023-09-29T23:08:12.562 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.559+0000 7fdeed811700 1 -- 172.21.15.121:0/2918564056 wait complete. 2023-09-29T23:08:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.560+0000 7fdeed811700 1 Processor -- start 2023-09-29T23:08:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.560+0000 7fdeed811700 1 -- start start 2023-09-29T23:08:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80630e0 0x7fdee809b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee809c700 0x7fdee80a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee80a7b10 con 0x7fdee80630e0 2023-09-29T23:08:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee809ea10 con 0x7fdee80b3cf0 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee81535b0 con 0x7fdee809c700 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80630e0 0x7fdee809b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80630e0 0x7fdee809b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45586/0 (socket says 172.21.15.121:45586) 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 -- 172.21.15.121:0/3894373834 learned_addr learned my addr 172.21.15.121:0/3894373834 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee67fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee809c700 0x7fdee80a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee77fe700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 -- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee809c700 msgr2=0x7fdee80a75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee809c700 0x7fdee80a75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 -- 172.21.15.121:0/3894373834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdee80b3cf0 msgr2=0x7fdee80a8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 -- 172.21.15.121:0/3894373834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc009480 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee77fe700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdee6ffd700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80630e0 0x7fdee809b040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00ec80 tx=0x7fdedc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeceffd700 1 -- 172.21.15.121:0/3894373834 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc005a00 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeceffd700 1 -- 172.21.15.121:0/3894373834 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc005b60 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee0009a20 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeceffd700 1 -- 172.21.15.121:0/3894373834 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc00c8a0 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.561+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee809b710 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.562+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee80a7cb0 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.563+0000 7fdeceffd700 1 -- 172.21.15.121:0/3894373834 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fdedc00cc50 con 0x7fdee80630e0 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.563+0000 7fdeceffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.564+0000 7fdeceffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.564+0000 7fdeceffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.564+0000 7fdeceffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.564+0000 7fdeceffd700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fded4098780 0x7fded409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.564+0000 7fdee67fc700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fded4098780 0x7fded409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.564+0000 7fdeceffd700 1 -- 172.21.15.121:0/3894373834 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fdedc0c0d00 con 0x7fdee80630e0 2023-09-29T23:08:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.564+0000 7fdee67fc700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fded4098780 0x7fded409ac30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdee81448a0 tx=0x7fded8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:12.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.567+0000 7fdeceffd700 1 -- 172.21.15.121:0/3894373834 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdedc08e320 con 0x7fdee80630e0 2023-09-29T23:08:12.788 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.787+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdee80a7cb0 con 0x7fdee80630e0 2023-09-29T23:08:12.788 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.788+0000 7fdeceffd700 1 -- 172.21.15.121:0/3894373834 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdee80a7cb0 con 0x7fdee80630e0 2023-09-29T23:08:12.789 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:12.789 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 10 mgrc shutdown 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fded4098780 msgr2=0x7fded409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fded4098780 0x7fded409ac30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdee81448a0 tx=0x7fded8009040 comp rx=0 tx=0).stop 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80630e0 msgr2=0x7fdee809b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80630e0 0x7fdee809b040 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00ec80 tx=0x7fdedc009220 comp rx=0 tx=0).stop 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 shutdown_connections 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fded4098780 0x7fded409ac30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.791 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdee80b3cf0 0x7fdee80a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.792 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdee80630e0 0x7fdee809b040 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.792 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 --2- 172.21.15.121:0/3894373834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdee809c700 0x7fdee80a75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:12.792 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 >> 172.21.15.121:0/3894373834 conn(0x7fdee8069e90 msgr2=0x7fdee80b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:12.792 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 shutdown_connections 2023-09-29T23:08:12.792 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:12.790+0000 7fdeed811700 1 -- 172.21.15.121:0/3894373834 wait complete. 2023-09-29T23:08:15.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:15.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.933+0000 7f5cdcc77700 1 Processor -- start 2023-09-29T23:08:15.934 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.933+0000 7f5cdcc77700 1 -- start start 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cdcc77700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5cd8141440 0x7f5cd80a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cdcc77700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd80630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cdcc77700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd8141810 con 0x7f5cd80b3cf0 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cdcc77700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd809a7f0 con 0x7f5cd8141440 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cdcc77700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80a4df0 con 0x7f5cd809b770 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd80630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd80630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:51538/0 (socket says 172.21.15.121:51538) 2023-09-29T23:08:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd659c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5cd8141440 0x7f5cd80a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 -- 172.21.15.121:0/702274800 learned_addr learned my addr 172.21.15.121:0/702274800 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 -- 172.21.15.121:0/702274800 >> v2:172.21.15.121:3300/0 conn(0x7f5cd8141440 msgr2=0x7f5cd80a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 --2- 172.21.15.121:0/702274800 >> v2:172.21.15.121:3300/0 conn(0x7f5cd8141440 0x7f5cd80a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 -- 172.21.15.121:0/702274800 >> v1:172.21.15.121:6789/0 conn(0x7f5cd80b3cf0 legacy=0x7f5cd8143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 -- 172.21.15.121:0/702274800 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd80a50b0 con 0x7f5cd809b770 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd659c700 1 --2- 172.21.15.121:0/702274800 >> v2:172.21.15.121:3300/0 conn(0x7f5cd8141440 0x7f5cd80a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd5d9b700 1 --2- 172.21.15.121:0/702274800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd80630e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0099b0 tx=0x7f5ccc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae87a9fa5f351220 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd559a700 1 -- 172.21.15.121:0/702274800 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc009ea0 con 0x7f5cd809b770 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd559a700 1 -- 172.21.15.121:0/702274800 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc00b860 con 0x7f5cd809b770 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.934+0000 7f5cd559a700 1 -- 172.21.15.121:0/702274800 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc00bc20 con 0x7f5cd809b770 2023-09-29T23:08:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 -- 172.21.15.121:0/702274800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 msgr2=0x7f5cd80630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:15.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/702274800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd80630e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc0099b0 tx=0x7f5ccc009250 comp rx=0 tx=0).stop 2023-09-29T23:08:15.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 -- 172.21.15.121:0/702274800 shutdown_connections 2023-09-29T23:08:15.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/702274800 >> v2:172.21.15.121:3300/0 conn(0x7f5cd8141440 0x7f5cd80a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:15.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/702274800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd80630e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:15.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 -- 172.21.15.121:0/702274800 >> 172.21.15.121:0/702274800 conn(0x7f5cd8069e90 msgr2=0x7f5cd80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:15.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 -- 172.21.15.121:0/702274800 shutdown_connections 2023-09-29T23:08:15.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.935+0000 7f5cdcc77700 1 -- 172.21.15.121:0/702274800 wait complete. 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.936+0000 7f5cdcc77700 1 Processor -- start 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.936+0000 7f5cdcc77700 1 -- start start 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd8139190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cd80b3cf0 0x7f5cd80a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cd8141440 0x7f5cd80a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd809f090 con 0x7f5cd8141440 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80a2360 con 0x7f5cd80b3cf0 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd8142470 con 0x7f5cd809b770 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd8139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:15.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd8139190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:51548/0 (socket says 172.21.15.121:51548) 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd659c700 1 -- 172.21.15.121:0/4273107188 learned_addr learned my addr 172.21.15.121:0/4273107188 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd6d9d700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cd8141440 0x7f5cd80a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd5d9b700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cd80b3cf0 0x7f5cd80a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd6d9d700 1 -- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 msgr2=0x7f5cd8139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd6d9d700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd8139190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd6d9d700 1 -- 172.21.15.121:0/4273107188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cd80b3cf0 msgr2=0x7f5cd80a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd6d9d700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cd80b3cf0 0x7f5cd80a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd6d9d700 1 -- 172.21.15.121:0/4273107188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc80093c0 con 0x7f5cd8141440 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd659c700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd8139190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:15.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cd6d9d700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cd8141440 0x7f5cd80a1c30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80b40f0 tx=0x7f5cd0001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cc67fc700 1 -- 172.21.15.121:0/4273107188 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0012630 con 0x7f5cd8141440 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cc67fc700 1 -- 172.21.15.121:0/4273107188 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd0012790 con 0x7f5cd8141440 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc0094f0 con 0x7f5cd8141440 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cc67fc700 1 -- 172.21.15.121:0/4273107188 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0012b50 con 0x7f5cd8141440 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.937+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd80a25c0 con 0x7f5cd8141440 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.938+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd806ab60 con 0x7f5cd8141440 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.939+0000 7f5cc67fc700 1 -- 172.21.15.121:0/4273107188 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5cd001a3a0 con 0x7f5cd8141440 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.939+0000 7f5cc67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.940+0000 7f5cc67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.940+0000 7f5cc67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.940+0000 7f5cc67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.940+0000 7f5cc67fc700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5cb8098780 0x7f5cb809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.940+0000 7f5cd659c700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5cb8098780 0x7f5cb809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.940+0000 7f5cc67fc700 1 -- 172.21.15.121:0/4273107188 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f5cd00c0650 con 0x7f5cd8141440 2023-09-29T23:08:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.940+0000 7f5cd659c700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5cb8098780 0x7f5cb809ac30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009fd0 tx=0x7f5cc8009120 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:15.944+0000 7f5cc67fc700 1 -- 172.21.15.121:0/4273107188 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5cd008dc70 con 0x7f5cd8141440 2023-09-29T23:08:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.166+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5cd8071660 con 0x7f5cd8141440 2023-09-29T23:08:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.166+0000 7f5cc67fc700 1 -- 172.21.15.121:0/4273107188 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5cd0091290 con 0x7f5cd8141440 2023-09-29T23:08:16.168 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:16.168 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 10 mgrc shutdown 2023-09-29T23:08:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5cb8098780 msgr2=0x7f5cb809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5cb8098780 0x7f5cb809ac30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009fd0 tx=0x7f5cc8009120 comp rx=0 tx=0).stop 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cd8141440 msgr2=0x7f5cd80a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cd8141440 0x7f5cd80a1c30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80b40f0 tx=0x7f5cd0001690 comp rx=0 tx=0).stop 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 shutdown_connections 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5cb8098780 0x7f5cb809ac30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cd8141440 0x7f5cd80a1c30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5cd809b770 0x7f5cd8139190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 --2- 172.21.15.121:0/4273107188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cd80b3cf0 0x7f5cd80a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 >> 172.21.15.121:0/4273107188 conn(0x7f5cd8069e90 msgr2=0x7f5cd813c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.168+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 shutdown_connections 2023-09-29T23:08:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:16.169+0000 7f5cdcc77700 1 -- 172.21.15.121:0/4273107188 wait complete. 2023-09-29T23:08:19.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:19.311 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.310+0000 7fcf3826d700 1 Processor -- start 2023-09-29T23:08:19.311 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.310+0000 7fcf3826d700 1 -- start start 2023-09-29T23:08:19.313 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3826d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcf301408c0 0x7fcf30140cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:19.313 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3826d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf30147b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:19.313 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3826d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3009dd50 con 0x7fcf300b39c0 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3826d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf300a63c0 con 0x7fcf301408c0 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3826d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3009efd0 con 0x7fcf30141200 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3680a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf30147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf36009700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcf300b39c0 0x7fcf300a5980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35358/0 (socket says 172.21.15.121:35358) 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf36009700 1 -- 172.21.15.121:0/2041193225 learned_addr learned my addr 172.21.15.121:0/2041193225 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf35808700 1 --2- 172.21.15.121:0/2041193225 >> v2:172.21.15.123:3300/0 conn(0x7fcf301408c0 0x7fcf30140cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf35007700 1 -- 172.21.15.121:0/2041193225 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705201587 0 0) 0x7fcf3009dd50 con 0x7fcf300b39c0 2023-09-29T23:08:19.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3680a700 1 -- 172.21.15.121:0/2041193225 >> v2:172.21.15.123:3300/0 conn(0x7fcf301408c0 msgr2=0x7fcf30140cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3680a700 1 --2- 172.21.15.121:0/2041193225 >> v2:172.21.15.123:3300/0 conn(0x7fcf301408c0 0x7fcf30140cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3680a700 1 -- 172.21.15.121:0/2041193225 >> v1:172.21.15.121:6789/0 conn(0x7fcf300b39c0 legacy=0x7fcf300a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3680a700 1 -- 172.21.15.121:0/2041193225 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3009e6d0 con 0x7fcf30141200 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.311+0000 7fcf3680a700 1 --2- 172.21.15.121:0/2041193225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf30147b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009e60 tx=0x7fcf28009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d48b270725354787 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf35007700 1 -- 172.21.15.121:0/2041193225 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2800e070 con 0x7fcf30141200 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf35007700 1 -- 172.21.15.121:0/2041193225 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2800bbf0 con 0x7fcf30141200 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf35007700 1 -- 172.21.15.121:0/2041193225 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28012430 con 0x7fcf30141200 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 -- 172.21.15.121:0/2041193225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 msgr2=0x7fcf30147b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2041193225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf30147b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009e60 tx=0x7fcf28009780 comp rx=0 tx=0).stop 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 -- 172.21.15.121:0/2041193225 shutdown_connections 2023-09-29T23:08:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2041193225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf30147b10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2041193225 >> v2:172.21.15.123:3300/0 conn(0x7fcf301408c0 0x7fcf30140cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 -- 172.21.15.121:0/2041193225 >> 172.21.15.121:0/2041193225 conn(0x7fcf30069e90 msgr2=0x7fcf300b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 -- 172.21.15.121:0/2041193225 shutdown_connections 2023-09-29T23:08:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.312+0000 7fcf3826d700 1 -- 172.21.15.121:0/2041193225 wait complete. 2023-09-29T23:08:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.313+0000 7fcf3826d700 1 Processor -- start 2023-09-29T23:08:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.313+0000 7fcf3826d700 1 -- start start 2023-09-29T23:08:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf3826d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf300b39c0 0x7fcf300a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf3826d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf301408c0 0x7fcf3013c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf3826d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf3006dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf3826d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3006f2c0 con 0x7fcf300b39c0 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf3826d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30140710 con 0x7fcf301408c0 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf3826d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3006e330 con 0x7fcf30141200 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf300b39c0 0x7fcf300a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf300b39c0 0x7fcf300a2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50638/0 (socket says 172.21.15.121:50638) 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 -- 172.21.15.121:0/2534060710 learned_addr learned my addr 172.21.15.121:0/2534060710 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf3680a700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf3006dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf35808700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf301408c0 0x7fcf3013c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 -- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 msgr2=0x7fcf3006dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf3006dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 -- 172.21.15.121:0/2534060710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf301408c0 msgr2=0x7fcf3013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf301408c0 0x7fcf3013c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 -- 172.21.15.121:0/2534060710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28009a20 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf35808700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf301408c0 0x7fcf3013c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.314+0000 7fcf36009700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf300b39c0 0x7fcf300a2120 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcf20001360 tx=0x7fcf20007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.315+0000 7fcf267fc700 1 -- 172.21.15.121:0/2534060710 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2000ee50 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.315+0000 7fcf267fc700 1 -- 172.21.15.121:0/2534060710 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf20007300 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.315+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3006e5d0 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.315+0000 7fcf267fc700 1 -- 172.21.15.121:0/2534060710 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf200057f0 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.315+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf300a2ae0 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.316+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3006ab60 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf267fc700 1 -- 172.21.15.121:0/2534060710 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcf2000c7e0 con 0x7fcf300b39c0 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf267fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf267fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf267fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf267fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf267fc700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcf14098780 0x7fcf1409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf267fc700 1 -- 172.21.15.121:0/2534060710 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fcf200c2be0 con 0x7fcf300b39c0 2023-09-29T23:08:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.317+0000 7fcf35808700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcf14098780 0x7fcf1409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.318+0000 7fcf35808700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcf14098780 0x7fcf1409ac30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009d80 tx=0x7fcf2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.320+0000 7fcf267fc700 1 -- 172.21.15.121:0/2534060710 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf20090200 con 0x7fcf300b39c0 2023-09-29T23:08:19.541 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.540+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf30071660 con 0x7fcf300b39c0 2023-09-29T23:08:19.541 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.540+0000 7fcf267fc700 1 -- 172.21.15.121:0/2534060710 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcf20015080 con 0x7fcf300b39c0 2023-09-29T23:08:19.541 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:19.542 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 10 mgrc shutdown 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcf14098780 msgr2=0x7fcf1409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcf14098780 0x7fcf1409ac30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009d80 tx=0x7fcf2c008040 comp rx=0 tx=0).stop 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf300b39c0 msgr2=0x7fcf300a2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf300b39c0 0x7fcf300a2120 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcf20001360 tx=0x7fcf20007d10 comp rx=0 tx=0).stop 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 shutdown_connections 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcf14098780 0x7fcf1409ac30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf30141200 0x7fcf3006dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf300b39c0 0x7fcf300a2120 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 --2- 172.21.15.121:0/2534060710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf301408c0 0x7fcf3013c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.542+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 >> 172.21.15.121:0/2534060710 conn(0x7fcf30069e90 msgr2=0x7fcf300630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.543+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 shutdown_connections 2023-09-29T23:08:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:19.543+0000 7fcf3826d700 1 -- 172.21.15.121:0/2534060710 wait complete. 2023-09-29T23:08:22.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:22.691 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.683+0000 7f5543ee2700 1 Processor -- start 2023-09-29T23:08:22.691 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.683+0000 7f5543ee2700 1 -- start start 2023-09-29T23:08:22.691 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5543ee2700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f553c0630e0 0x7f553c09c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5543ee2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5543ee2700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c09e600 con 0x7f553c09f580 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5543ee2700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553c0a2d20 con 0x7f553c0630e0 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5543ee2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c09d8d0 con 0x7f553c0634b0 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f554147d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f554247f700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f553c09f580 0x7f553c09caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35360/0 (socket says 172.21.15.121:35360) 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f554247f700 1 -- 172.21.15.121:0/1858466471 learned_addr learned my addr 172.21.15.121:0/1858466471 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5541c7e700 1 --2- 172.21.15.121:0/1858466471 >> v2:172.21.15.123:3300/0 conn(0x7f553c0630e0 0x7f553c09c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5540c7c700 1 -- 172.21.15.121:0/1858466471 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544772916 0 0) 0x7f553c09e600 con 0x7f553c09f580 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.684+0000 7f5540c7c700 1 -- 172.21.15.121:0/1858466471 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5528003570 con 0x7f553c09f580 2023-09-29T23:08:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f554147d700 1 -- 172.21.15.121:0/1858466471 >> v2:172.21.15.123:3300/0 conn(0x7f553c0630e0 msgr2=0x7f553c09c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f554147d700 1 --2- 172.21.15.121:0/1858466471 >> v2:172.21.15.123:3300/0 conn(0x7f553c0630e0 0x7f553c09c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f554147d700 1 -- 172.21.15.121:0/1858466471 >> v1:172.21.15.121:6789/0 conn(0x7f553c09f580 legacy=0x7f553c09caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f554147d700 1 -- 172.21.15.121:0/1858466471 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c09db70 con 0x7f553c0634b0 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f554147d700 1 --2- 172.21.15.121:0/1858466471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c09c5b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f55380098e0 tx=0x7f55380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d457bab8c5ec0ec7 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5540c7c700 1 -- 172.21.15.121:0/1858466471 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538011630 con 0x7f553c0634b0 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5540c7c700 1 -- 172.21.15.121:0/1858466471 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5538011790 con 0x7f553c0634b0 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5540c7c700 1 -- 172.21.15.121:0/1858466471 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538011b30 con 0x7f553c0634b0 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5543ee2700 1 -- 172.21.15.121:0/1858466471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 msgr2=0x7f553c09c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5543ee2700 1 --2- 172.21.15.121:0/1858466471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c09c5b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f55380098e0 tx=0x7f55380091a0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5543ee2700 1 -- 172.21.15.121:0/1858466471 shutdown_connections 2023-09-29T23:08:22.693 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5543ee2700 1 --2- 172.21.15.121:0/1858466471 >> v2:172.21.15.123:3300/0 conn(0x7f553c0630e0 0x7f553c09c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5543ee2700 1 --2- 172.21.15.121:0/1858466471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c09c5b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.685+0000 7f5543ee2700 1 -- 172.21.15.121:0/1858466471 >> 172.21.15.121:0/1858466471 conn(0x7f553c069e90 msgr2=0x7f553c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.686+0000 7f5543ee2700 1 -- 172.21.15.121:0/1858466471 shutdown_connections 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.686+0000 7f5543ee2700 1 -- 172.21.15.121:0/1858466471 wait complete. 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.686+0000 7f5543ee2700 1 Processor -- start 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.686+0000 7f5543ee2700 1 -- start start 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5543ee2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f553c0630e0 0x7f553c0a7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5543ee2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c0a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:22.694 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5543ee2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f553c09f580 0x7f553c1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5543ee2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c09af80 con 0x7f553c0630e0 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5543ee2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c151300 con 0x7f553c09f580 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5543ee2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c14de10 con 0x7f553c0634b0 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5541c7e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f553c0630e0 0x7f553c0a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c0a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c0a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53754/0 (socket says 172.21.15.121:53754) 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5541c7e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f553c0630e0 0x7f553c0a7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50652/0 (socket says 172.21.15.121:50652) 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 -- 172.21.15.121:0/2356671601 learned_addr learned my addr 172.21.15.121:0/2356671601 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 -- 172.21.15.121:0/2356671601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f553c09f580 msgr2=0x7f553c1517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554247f700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f553c09f580 0x7f553c1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f553c09f580 0x7f553c1517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 -- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f553c0630e0 msgr2=0x7f553c0a7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f553c0630e0 0x7f553c0a7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f554147d700 1 -- 172.21.15.121:0/2356671601 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5538009420 con 0x7f553c0634b0 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.687+0000 7f5541c7e700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f553c0630e0 0x7f553c0a7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.688+0000 7f554147d700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c0a45a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f553c09e790 tx=0x7f5538017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.688+0000 7f5531ffb700 1 -- 172.21.15.121:0/2356671601 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538011b10 con 0x7f553c0634b0 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.688+0000 7f5531ffb700 1 -- 172.21.15.121:0/2356671601 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5538000cc0 con 0x7f553c0634b0 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.688+0000 7f5531ffb700 1 -- 172.21.15.121:0/2356671601 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538025620 con 0x7f553c0634b0 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.688+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c0a4ae0 con 0x7f553c0634b0 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.688+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c151d00 con 0x7f553c0634b0 2023-09-29T23:08:22.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.689+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553c06ab60 con 0x7f553c0634b0 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.690+0000 7f5531ffb700 1 -- 172.21.15.121:0/2356671601 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5538025a10 con 0x7f553c0634b0 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.690+0000 7f5531ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.690+0000 7f5531ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.690+0000 7f5531ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.690+0000 7f5531ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.690+0000 7f5531ffb700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5520098800 0x7f552009acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.691+0000 7f5541c7e700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5520098800 0x7f552009acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:22.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.691+0000 7f5531ffb700 1 -- 172.21.15.121:0/2356671601 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f55380c1d10 con 0x7f553c0634b0 2023-09-29T23:08:22.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.691+0000 7f5541c7e700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5520098800 0x7f552009acb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f552c011460 tx=0x7f552c009120 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:22.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.695+0000 7f5531ffb700 1 -- 172.21.15.121:0/2356671601 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f553808f320 con 0x7f553c0634b0 2023-09-29T23:08:22.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.912+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f553c071660 con 0x7f553c0634b0 2023-09-29T23:08:22.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.912+0000 7f5531ffb700 1 -- 172.21.15.121:0/2356671601 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5538092940 con 0x7f553c0634b0 2023-09-29T23:08:22.914 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:22.914 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.914+0000 7f5543ee2700 10 mgrc shutdown 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.914+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5520098800 msgr2=0x7f552009acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5520098800 0x7f552009acb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f552c011460 tx=0x7f552c009120 comp rx=0 tx=0).stop 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 msgr2=0x7f553c0a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c0a45a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f553c09e790 tx=0x7f5538017000 comp rx=0 tx=0).stop 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 shutdown_connections 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5520098800 0x7f552009acb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f553c09f580 0x7f553c1517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f553c0630e0 0x7f553c0a7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 --2- 172.21.15.121:0/2356671601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f553c0634b0 0x7f553c0a45a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:22.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 >> 172.21.15.121:0/2356671601 conn(0x7f553c069e90 msgr2=0x7f553c0b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:22.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 shutdown_connections 2023-09-29T23:08:22.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:22.915+0000 7f5543ee2700 1 -- 172.21.15.121:0/2356671601 wait complete. 2023-09-29T23:08:25.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:26.057 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.057+0000 7ff12c2e1700 1 Processor -- start 2023-09-29T23:08:26.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.057+0000 7ff12c2e1700 1 -- start start 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12c2e1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff124141440 0x7ff1240a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12c2e1700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff1240b3cf0 0x7ff12414b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12c2e1700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff124141810 con 0x7ff12409b770 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12c2e1700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff12409a7f0 con 0x7ff124141440 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12c2e1700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1240a4df0 con 0x7ff1240b3cf0 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff124141440 0x7ff1240a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff124141440 0x7ff1240a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50662/0 (socket says 172.21.15.121:50662) 2023-09-29T23:08:26.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 -- 172.21.15.121:0/1839707782 learned_addr learned my addr 172.21.15.121:0/1839707782 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 -- 172.21.15.121:0/1839707782 >> v2:172.21.15.123:3300/0 conn(0x7ff1240b3cf0 msgr2=0x7ff12414b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 --2- 172.21.15.121:0/1839707782 >> v2:172.21.15.123:3300/0 conn(0x7ff1240b3cf0 0x7ff12414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 -- 172.21.15.121:0/1839707782 >> v1:172.21.15.123:6789/0 conn(0x7ff12409b770 legacy=0x7ff1240630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 -- 172.21.15.121:0/1839707782 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240a50b0 con 0x7ff124141440 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12a07d700 1 --2- 172.21.15.121:0/1839707782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff124141440 0x7ff1240a45a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff12406b4e0 tx=0x7ff114009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60349397b6badb38 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12907b700 1 -- 172.21.15.121:0/1839707782 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114005c10 con 0x7ff124141440 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12907b700 1 -- 172.21.15.121:0/1839707782 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114005d70 con 0x7ff124141440 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.058+0000 7ff12907b700 1 -- 172.21.15.121:0/1839707782 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11400b840 con 0x7ff124141440 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 -- 172.21.15.121:0/1839707782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff124141440 msgr2=0x7ff1240a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/1839707782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff124141440 0x7ff1240a45a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff12406b4e0 tx=0x7ff114009220 comp rx=0 tx=0).stop 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 -- 172.21.15.121:0/1839707782 shutdown_connections 2023-09-29T23:08:26.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/1839707782 >> v2:172.21.15.123:3300/0 conn(0x7ff1240b3cf0 0x7ff12414b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/1839707782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff124141440 0x7ff1240a45a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 -- 172.21.15.121:0/1839707782 >> 172.21.15.121:0/1839707782 conn(0x7ff124069e90 msgr2=0x7ff1240b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 -- 172.21.15.121:0/1839707782 shutdown_connections 2023-09-29T23:08:26.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.059+0000 7ff12c2e1700 1 -- 172.21.15.121:0/1839707782 wait complete. 2023-09-29T23:08:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 Processor -- start 2023-09-29T23:08:26.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 -- start start 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff12409b770 0x7ff1240a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1240b3cf0 0x7ff1240a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff124141440 0x7ff1240a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240a2c70 con 0x7ff12409b770 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240a30c0 con 0x7ff1240b3cf0 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12c2e1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff124150580 con 0x7ff124141440 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12a07d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff12409b770 0x7ff1240a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12a87e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff124141440 0x7ff1240a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.060+0000 7ff12a07d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff12409b770 0x7ff1240a1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50672/0 (socket says 172.21.15.121:50672) 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12a07d700 1 -- 172.21.15.121:0/2209202588 learned_addr learned my addr 172.21.15.121:0/2209202588 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:26.065 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12987c700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1240b3cf0 0x7ff1240a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12a07d700 1 -- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff124141440 msgr2=0x7ff1240a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12a07d700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff124141440 0x7ff1240a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12a07d700 1 -- 172.21.15.121:0/2209202588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1240b3cf0 msgr2=0x7ff1240a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12a07d700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1240b3cf0 0x7ff1240a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12a07d700 1 -- 172.21.15.121:0/2209202588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff114009480 con 0x7ff12409b770 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12a07d700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff12409b770 0x7ff1240a1fe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff12406ad60 tx=0x7ff114015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff11a7fc700 1 -- 172.21.15.121:0/2209202588 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1140119b0 con 0x7ff12409b770 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff11a7fc700 1 -- 172.21.15.121:0/2209202588 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114011b30 con 0x7ff12409b770 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1240a10f0 con 0x7ff12409b770 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff11a7fc700 1 -- 172.21.15.121:0/2209202588 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11401d430 con 0x7ff12409b770 2023-09-29T23:08:26.066 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.061+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1241446c0 con 0x7ff12409b770 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.062+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1240a2de0 con 0x7ff12409b770 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.063+0000 7ff11a7fc700 1 -- 172.21.15.121:0/2209202588 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff11401d860 con 0x7ff12409b770 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.063+0000 7ff11a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.063+0000 7ff11a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.063+0000 7ff11a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.063+0000 7ff11a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.063+0000 7ff11a7fc700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff108098780 0x7ff10809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.064+0000 7ff12987c700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff108098780 0x7ff10809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.064+0000 7ff11a7fc700 1 -- 172.21.15.121:0/2209202588 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7ff1140c24c0 con 0x7ff12409b770 2023-09-29T23:08:26.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.064+0000 7ff12987c700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff108098780 0x7ff10809ac30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff12409bf10 tx=0x7ff120008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:26.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.067+0000 7ff11a7fc700 1 -- 172.21.15.121:0/2209202588 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff11405d5a0 con 0x7ff12409b770 2023-09-29T23:08:26.289 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.289+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff12406ab60 con 0x7ff12409b770 2023-09-29T23:08:26.290 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.289+0000 7ff11a7fc700 1 -- 172.21.15.121:0/2209202588 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff11408fad0 con 0x7ff12409b770 2023-09-29T23:08:26.290 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:26.290 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:26.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 10 mgrc shutdown 2023-09-29T23:08:26.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff108098780 msgr2=0x7ff10809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:26.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff108098780 0x7ff10809ac30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff12409bf10 tx=0x7ff120008040 comp rx=0 tx=0).stop 2023-09-29T23:08:26.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff12409b770 msgr2=0x7ff1240a1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff12409b770 0x7ff1240a1fe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff12406ad60 tx=0x7ff114015000 comp rx=0 tx=0).stop 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 shutdown_connections 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff108098780 0x7ff10809ac30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff124141440 0x7ff1240a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff12409b770 0x7ff1240a1fe0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 --2- 172.21.15.121:0/2209202588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1240b3cf0 0x7ff1240a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 >> 172.21.15.121:0/2209202588 conn(0x7ff124069e90 msgr2=0x7ff12413f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 shutdown_connections 2023-09-29T23:08:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:26.291+0000 7ff12c2e1700 1 -- 172.21.15.121:0/2209202588 wait complete. 2023-09-29T23:08:29.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:29.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4819249700 1 Processor -- start 2023-09-29T23:08:29.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4819249700 1 -- start start 2023-09-29T23:08:29.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4819249700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4814141440 0x7f48140a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:29.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4819249700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f48140b3cf0 0x7f481414b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:29.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4819249700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4814141810 con 0x7f481409b770 2023-09-29T23:08:29.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4819249700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481409a7f0 con 0x7f4814141440 2023-09-29T23:08:29.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4819249700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48140a4df0 con 0x7f48140b3cf0 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4812d9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4814141440 0x7f48140a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4812d9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4814141440 0x7f48140a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41092/0 (socket says 172.21.15.121:41092) 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.432+0000 7f4812d9d700 1 -- 172.21.15.121:0/3430725297 learned_addr learned my addr 172.21.15.121:0/3430725297 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f481359e700 1 --2- 172.21.15.121:0/3430725297 >> v2:172.21.15.123:3300/0 conn(0x7f48140b3cf0 0x7f481414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4812d9d700 1 -- 172.21.15.121:0/3430725297 >> v2:172.21.15.123:3300/0 conn(0x7f48140b3cf0 msgr2=0x7f481414b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4812d9d700 1 --2- 172.21.15.121:0/3430725297 >> v2:172.21.15.123:3300/0 conn(0x7f48140b3cf0 0x7f481414b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4812d9d700 1 -- 172.21.15.121:0/3430725297 >> v1:172.21.15.123:6789/0 conn(0x7f481409b770 legacy=0x7f48140630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4812d9d700 1 -- 172.21.15.121:0/3430725297 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48140a50b0 con 0x7f4814141440 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4812d9d700 1 --2- 172.21.15.121:0/3430725297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48140a45a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f481406b4e0 tx=0x7f4804009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6975894ba3212f server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4811d9b700 1 -- 172.21.15.121:0/3430725297 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804005c10 con 0x7f4814141440 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4811d9b700 1 -- 172.21.15.121:0/3430725297 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4804005d70 con 0x7f4814141440 2023-09-29T23:08:29.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.433+0000 7f4811d9b700 1 -- 172.21.15.121:0/3430725297 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480400b8a0 con 0x7f4814141440 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 -- 172.21.15.121:0/3430725297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 msgr2=0x7f48140a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 --2- 172.21.15.121:0/3430725297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48140a45a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f481406b4e0 tx=0x7f4804009220 comp rx=0 tx=0).stop 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 -- 172.21.15.121:0/3430725297 shutdown_connections 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 --2- 172.21.15.121:0/3430725297 >> v2:172.21.15.123:3300/0 conn(0x7f48140b3cf0 0x7f481414b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 --2- 172.21.15.121:0/3430725297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48140a45a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 -- 172.21.15.121:0/3430725297 >> 172.21.15.121:0/3430725297 conn(0x7f4814069e90 msgr2=0x7f48140b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 -- 172.21.15.121:0/3430725297 shutdown_connections 2023-09-29T23:08:29.437 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.434+0000 7f4819249700 1 -- 172.21.15.121:0/3430725297 wait complete. 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 Processor -- start 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 -- start start 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f481409b770 0x7f481409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48140b3cf0 0x7f48141437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48141508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48141432b0 con 0x7f4814141440 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481409e230 con 0x7f48140b3cf0 2023-09-29T23:08:29.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4819249700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4814144150 con 0x7f481409b770 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f4812d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f481409b770 0x7f481409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.435+0000 7f481359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48141508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48141508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41102/0 (socket says 172.21.15.121:41102) 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f4812d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f481409b770 0x7f481409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56748/0 (socket says 172.21.15.121:56748) 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 -- 172.21.15.121:0/194728489 learned_addr learned my addr 172.21.15.121:0/194728489 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481259c700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48140b3cf0 0x7f48141437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 -- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f481409b770 msgr2=0x7f481409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f481409b770 0x7f481409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.441 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 -- 172.21.15.121:0/194728489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48140b3cf0 msgr2=0x7f48141437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48140b3cf0 0x7f48141437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 -- 172.21.15.121:0/194728489 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804009480 con 0x7f4814141440 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f4812d9d700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f481409b770 0x7f481409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f481359e700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48141508c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f480c007ed0 tx=0x7f480c009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f47faffd700 1 -- 172.21.15.121:0/194728489 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c00eb90 con 0x7f4814141440 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f47faffd700 1 -- 172.21.15.121:0/194728489 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480c00ed10 con 0x7f4814141440 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f47faffd700 1 -- 172.21.15.121:0/194728489 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c014670 con 0x7f4814141440 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4814150470 con 0x7f4814141440 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.436+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48140a2c30 con 0x7f4814141440 2023-09-29T23:08:29.442 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.437+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481406ab60 con 0x7f4814141440 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.438+0000 7f47faffd700 1 -- 172.21.15.121:0/194728489 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f480c014aa0 con 0x7f4814141440 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.438+0000 7f47faffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.438+0000 7f47faffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.438+0000 7f47faffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.438+0000 7f47faffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.438+0000 7f47faffd700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4800098780 0x7f480009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.439+0000 7f4812d9d700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4800098780 0x7f480009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.439+0000 7f47faffd700 1 -- 172.21.15.121:0/194728489 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f480c0c0ab0 con 0x7f4814141440 2023-09-29T23:08:29.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.439+0000 7f4812d9d700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4800098780 0x7f480009ac30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4804007fd0 tx=0x7f4804016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:29.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.443+0000 7f47faffd700 1 -- 172.21.15.121:0/194728489 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f480c08e0d0 con 0x7f4814141440 2023-09-29T23:08:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.663+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4814071660 con 0x7f4814141440 2023-09-29T23:08:29.664 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.663+0000 7f47faffd700 1 -- 172.21.15.121:0/194728489 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f480c0916f0 con 0x7f4814141440 2023-09-29T23:08:29.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:29.664 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 10 mgrc shutdown 2023-09-29T23:08:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4800098780 msgr2=0x7f480009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4800098780 0x7f480009ac30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4804007fd0 tx=0x7f4804016000 comp rx=0 tx=0).stop 2023-09-29T23:08:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 msgr2=0x7f48141508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48141508c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f480c007ed0 tx=0x7f480c009880 comp rx=0 tx=0).stop 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 shutdown_connections 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4800098780 0x7f480009ac30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4814141440 0x7f48141508c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f481409b770 0x7f481409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 --2- 172.21.15.121:0/194728489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48140b3cf0 0x7f48141437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 >> 172.21.15.121:0/194728489 conn(0x7f4814069e90 msgr2=0x7f48140a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 shutdown_connections 2023-09-29T23:08:29.667 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:29.665+0000 7f4819249700 1 -- 172.21.15.121:0/194728489 wait complete. 2023-09-29T23:08:32.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:32.807 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efba77700 1 Processor -- start 2023-09-29T23:08:32.807 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efba77700 1 -- start start 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efba77700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7ef40b3cf0 0x7f7ef4146900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efba77700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef4148e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efba77700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef40a0620 con 0x7f7ef40630e0 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efba77700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef40a08e0 con 0x7f7ef40b3cf0 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efba77700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef409c8d0 con 0x7f7ef40a2180 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7efa014700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef4148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7ef9813700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7ef40630e0 0x7f7ef409c1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:41230/0 (socket says 172.21.15.121:41230) 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.807+0000 7f7ef9813700 1 -- 172.21.15.121:0/2390699644 learned_addr learned my addr 172.21.15.121:0/2390699644 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:08:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7ef9012700 1 --2- 172.21.15.121:0/2390699644 >> v2:172.21.15.123:3300/0 conn(0x7f7ef40b3cf0 0x7f7ef4146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7efa014700 1 -- 172.21.15.121:0/2390699644 >> v2:172.21.15.123:3300/0 conn(0x7f7ef40b3cf0 msgr2=0x7f7ef4146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7efa014700 1 --2- 172.21.15.121:0/2390699644 >> v2:172.21.15.123:3300/0 conn(0x7f7ef40b3cf0 0x7f7ef4146900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7efa014700 1 -- 172.21.15.121:0/2390699644 >> v1:172.21.15.121:6789/0 conn(0x7f7ef40630e0 legacy=0x7f7ef409c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7ef8811700 1 -- 172.21.15.121:0/2390699644 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412912044 0 0) 0x7f7ef40a0620 con 0x7f7ef40630e0 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7efa014700 1 -- 172.21.15.121:0/2390699644 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef40a62d0 con 0x7f7ef40a2180 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7efa014700 1 --2- 172.21.15.121:0/2390699644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef4148e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009e60 tx=0x7f7eec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7bbb699eab7044b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7ef8811700 1 -- 172.21.15.121:0/2390699644 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec00e070 con 0x7f7ef40a2180 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7ef8811700 1 -- 172.21.15.121:0/2390699644 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eec00bc30 con 0x7f7ef40a2180 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7ef8811700 1 -- 172.21.15.121:0/2390699644 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec012430 con 0x7f7ef40a2180 2023-09-29T23:08:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.808+0000 7f7efba77700 1 -- 172.21.15.121:0/2390699644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 msgr2=0x7f7ef4148e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:32.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.809+0000 7f7efba77700 1 --2- 172.21.15.121:0/2390699644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef4148e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009e60 tx=0x7f7eec009780 comp rx=0 tx=0).stop 2023-09-29T23:08:32.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.809+0000 7f7efba77700 1 -- 172.21.15.121:0/2390699644 shutdown_connections 2023-09-29T23:08:32.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.809+0000 7f7efba77700 1 --2- 172.21.15.121:0/2390699644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef4148e50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:32.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.809+0000 7f7efba77700 1 --2- 172.21.15.121:0/2390699644 >> v2:172.21.15.123:3300/0 conn(0x7f7ef40b3cf0 0x7f7ef4146900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:32.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.809+0000 7f7efba77700 1 -- 172.21.15.121:0/2390699644 >> 172.21.15.121:0/2390699644 conn(0x7f7ef4069e90 msgr2=0x7f7ef40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:32.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.809+0000 7f7efba77700 1 -- 172.21.15.121:0/2390699644 shutdown_connections 2023-09-29T23:08:32.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.809+0000 7f7efba77700 1 -- 172.21.15.121:0/2390699644 wait complete. 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 Processor -- start 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 -- start start 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef40630e0 0x7f7ef406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef41510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40b3cf0 0x7f7ef413f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4099280 con 0x7f7ef40630e0 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4151a30 con 0x7f7ef40b3cf0 2023-09-29T23:08:32.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efba77700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4151cf0 con 0x7f7ef40a2180 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7ef9813700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef40630e0 0x7f7ef406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7ef9012700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef41510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7ef9012700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef41510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56762/0 (socket says 172.21.15.121:56762) 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7ef9813700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef40630e0 0x7f7ef406eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41110/0 (socket says 172.21.15.121:41110) 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7ef9012700 1 -- 172.21.15.121:0/949384058 learned_addr learned my addr 172.21.15.121:0/949384058 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.810+0000 7f7efa014700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40b3cf0 0x7f7ef413f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ef9813700 1 -- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 msgr2=0x7f7ef41510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ef9813700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef41510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ef9813700 1 -- 172.21.15.121:0/949384058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40b3cf0 msgr2=0x7f7ef413f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ef9813700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40b3cf0 0x7f7ef413f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:32.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ef9813700 1 -- 172.21.15.121:0/949384058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec009a20 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7efa014700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40b3cf0 0x7f7ef413f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ef9813700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef40630e0 0x7f7ef406eaa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c4e0 tx=0x7f7ee4007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ee9ffb700 1 -- 172.21.15.121:0/949384058 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40073b0 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ee9ffb700 1 -- 172.21.15.121:0/949384058 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee4007510 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef4096ad0 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7ee9ffb700 1 -- 172.21.15.121:0/949384058 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4005a40 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.811+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef409b780 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.812+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef406ab60 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.813+0000 7f7ee9ffb700 1 -- 172.21.15.121:0/949384058 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7ee401b3b0 con 0x7f7ef40630e0 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.813+0000 7f7ee9ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:32.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.814+0000 7f7ee9ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.814+0000 7f7ee9ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.814+0000 7f7ee9ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.814+0000 7f7ee9ffb700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7ed8098780 0x7f7ed809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.814+0000 7f7ef9012700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7ed8098780 0x7f7ed809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.814+0000 7f7ee9ffb700 1 -- 172.21.15.121:0/949384058 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f7ee40bfd80 con 0x7f7ef40630e0 2023-09-29T23:08:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.814+0000 7f7ef9012700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7ed8098780 0x7f7ed809ac30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ef41418a0 tx=0x7f7ef0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:32.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:32.818+0000 7f7ee9ffb700 1 -- 172.21.15.121:0/949384058 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ee408d390 con 0x7f7ef40630e0 2023-09-29T23:08:33.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.039+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7ef4071660 con 0x7f7ef40630e0 2023-09-29T23:08:33.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.039+0000 7f7ee9ffb700 1 -- 172.21.15.121:0/949384058 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7ee4017100 con 0x7f7ef40630e0 2023-09-29T23:08:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:33.040 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 10 mgrc shutdown 2023-09-29T23:08:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7ed8098780 msgr2=0x7f7ed809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7ed8098780 0x7f7ed809ac30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ef41418a0 tx=0x7f7ef0008040 comp rx=0 tx=0).stop 2023-09-29T23:08:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef40630e0 msgr2=0x7f7ef406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef40630e0 0x7f7ef406eaa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c4e0 tx=0x7f7ee4007dc0 comp rx=0 tx=0).stop 2023-09-29T23:08:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 shutdown_connections 2023-09-29T23:08:33.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7ed8098780 0x7f7ed809ac30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40b3cf0 0x7f7ef413f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef40630e0 0x7f7ef406eaa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 --2- 172.21.15.121:0/949384058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef40a2180 0x7f7ef41510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 >> 172.21.15.121:0/949384058 conn(0x7f7ef4069e90 msgr2=0x7f7ef4143e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 shutdown_connections 2023-09-29T23:08:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:33.041+0000 7f7efba77700 1 -- 172.21.15.121:0/949384058 wait complete. 2023-09-29T23:08:36.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac7ac61700 1 Processor -- start 2023-09-29T23:08:36.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac7ac61700 1 -- start start 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac7ac61700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fac7409e320 0x7fac74141c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac7ac61700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e720 0x7fac741421c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac7ac61700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7406b300 con 0x7fac7406af30 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac7ac61700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac74142700 con 0x7fac7409e320 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac7ac61700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7413e380 con 0x7fac7409e720 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac73fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fac7409e320 0x7fac74141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac791fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e720 0x7fac741421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac791fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e720 0x7fac741421c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56774/0 (socket says 172.21.15.121:56774) 2023-09-29T23:08:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac789fd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fac7406af30 0x7fac7413ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:41232/0 (socket says 172.21.15.121:41232) 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac73fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fac7409e320 0x7fac74141c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41112/0 (socket says 172.21.15.121:41112) 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.185+0000 7fac791fe700 1 -- 172.21.15.121:0/1292564017 learned_addr learned my addr 172.21.15.121:0/1292564017 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac73fff700 1 -- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e720 msgr2=0x7fac741421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac73fff700 1 --2- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e720 0x7fac741421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac73fff700 1 -- 172.21.15.121:0/1292564017 >> v1:172.21.15.121:6789/0 conn(0x7fac7406af30 legacy=0x7fac7413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac73fff700 1 -- 172.21.15.121:0/1292564017 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7413e5e0 con 0x7fac7409e320 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac737fe700 1 -- 172.21.15.121:0/1292564017 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242088361 0 0) 0x7fac7406b300 con 0x7fac7406af30 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac791fe700 1 --2- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e720 0x7fac741421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac73fff700 1 --2- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7409e320 0x7fac74141c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fac600098e0 tx=0x7fac600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edd653616cc41b2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac737fe700 1 -- 172.21.15.121:0/1292564017 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6000b640 con 0x7fac7409e320 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac737fe700 1 -- 172.21.15.121:0/1292564017 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6000b7a0 con 0x7fac7409e320 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.186+0000 7fac737fe700 1 -- 172.21.15.121:0/1292564017 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6000bb40 con 0x7fac7409e320 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 -- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7409e320 msgr2=0x7fac74141c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 --2- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7409e320 0x7fac74141c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fac600098e0 tx=0x7fac600091a0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 -- 172.21.15.121:0/1292564017 shutdown_connections 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 --2- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e720 0x7fac741421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 --2- 172.21.15.121:0/1292564017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7409e320 0x7fac74141c80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 -- 172.21.15.121:0/1292564017 >> 172.21.15.121:0/1292564017 conn(0x7fac74067440 msgr2=0x7fac74067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 -- 172.21.15.121:0/1292564017 shutdown_connections 2023-09-29T23:08:36.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 -- 172.21.15.121:0/1292564017 wait complete. 2023-09-29T23:08:36.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.187+0000 7fac7ac61700 1 Processor -- start 2023-09-29T23:08:36.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac7ac61700 1 -- start start 2023-09-29T23:08:36.192 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac7ac61700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7406af30 0x7fac741470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac7ac61700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e320 0x7fac7413c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac7ac61700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac7409e720 0x7fac74069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac7ac61700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac74095de0 con 0x7fac7406af30 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac7ac61700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7413d350 con 0x7fac7409e720 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac7ac61700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7413d610 con 0x7fac7409e320 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac789fd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7406af30 0x7fac741470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac73fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e320 0x7fac7413c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac789fd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7406af30 0x7fac741470c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41126/0 (socket says 172.21.15.121:41126) 2023-09-29T23:08:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac73fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e320 0x7fac7413c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56790/0 (socket says 172.21.15.121:56790) 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.188+0000 7fac789fd700 1 -- 172.21.15.121:0/4272116950 learned_addr learned my addr 172.21.15.121:0/4272116950 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac791fe700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac7409e720 0x7fac74069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac73fff700 1 -- 172.21.15.121:0/4272116950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac7409e720 msgr2=0x7fac74069c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac73fff700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac7409e720 0x7fac74069c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac73fff700 1 -- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7406af30 msgr2=0x7fac741470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac73fff700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7406af30 0x7fac741470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac73fff700 1 -- 172.21.15.121:0/4272116950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c009ae0 con 0x7fac7409e320 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac789fd700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7406af30 0x7fac741470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac791fe700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac7409e720 0x7fac74069c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:08:36.194 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac73fff700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e320 0x7fac7413c9d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac60016000 tx=0x7fac60017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac70ff9700 1 -- 172.21.15.121:0/4272116950 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60000d10 con 0x7fac7409e320 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac70ff9700 1 -- 172.21.15.121:0/4272116950 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6000bb20 con 0x7fac7409e320 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac60009420 con 0x7fac7409e320 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac70ff9700 1 -- 172.21.15.121:0/4272116950 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60011650 con 0x7fac7409e320 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.189+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac74068920 con 0x7fac7409e320 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.190+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac74071660 con 0x7fac7409e320 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.191+0000 7fac70ff9700 1 -- 172.21.15.121:0/4272116950 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fac60011a00 con 0x7fac7409e320 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.191+0000 7fac70ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.191+0000 7fac70ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:36.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.191+0000 7fac70ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:36.196 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.191+0000 7fac70ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:36.196 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.191+0000 7fac70ff9700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fac58098780 0x7fac5809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:36.196 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.192+0000 7fac70ff9700 1 -- 172.21.15.121:0/4272116950 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fac600c1a40 con 0x7fac7409e320 2023-09-29T23:08:36.196 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.192+0000 7fac789fd700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fac58098780 0x7fac5809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:36.196 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.192+0000 7fac789fd700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fac58098780 0x7fac5809ac30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fac68007fd0 tx=0x7fac68007e10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:36.196 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.196+0000 7fac70ff9700 1 -- 172.21.15.121:0/4272116950 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fac600c3e60 con 0x7fac7409e320 2023-09-29T23:08:36.418 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.418+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fac7413e7f0 con 0x7fac7409e320 2023-09-29T23:08:36.419 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.419+0000 7fac70ff9700 1 -- 172.21.15.121:0/4272116950 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fac60015100 con 0x7fac7409e320 2023-09-29T23:08:36.419 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:36.419 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 10 mgrc shutdown 2023-09-29T23:08:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fac58098780 msgr2=0x7fac5809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fac58098780 0x7fac5809ac30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fac68007fd0 tx=0x7fac68007e10 comp rx=0 tx=0).stop 2023-09-29T23:08:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e320 msgr2=0x7fac7413c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e320 0x7fac7413c9d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac60016000 tx=0x7fac60017000 comp rx=0 tx=0).stop 2023-09-29T23:08:36.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 shutdown_connections 2023-09-29T23:08:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fac58098780 0x7fac5809ac30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac7409e720 0x7fac74069c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac7406af30 0x7fac741470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 --2- 172.21.15.121:0/4272116950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac7409e320 0x7fac7413c9d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 >> 172.21.15.121:0/4272116950 conn(0x7fac74067440 msgr2=0x7fac740b28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 shutdown_connections 2023-09-29T23:08:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:36.421+0000 7fac7ac61700 1 -- 172.21.15.121:0/4272116950 wait complete. 2023-09-29T23:08:39.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.563+0000 7fe1f6fe5700 1 Processor -- start 2023-09-29T23:08:39.563 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.563+0000 7fe1f6fe5700 1 -- start start 2023-09-29T23:08:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.563+0000 7fe1f6fe5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f00630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.563+0000 7fe1f6fe5700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe1f00b3cf0 0x7fe1f014b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.563+0000 7fe1f6fe5700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f0141810 con 0x7fe1f0141440 2023-09-29T23:08:39.565 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.563+0000 7fe1f6fe5700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f009a7f0 con 0x7fe1f00b3cf0 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.563+0000 7fe1f6fe5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f00a4df0 con 0x7fe1f009b770 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1f5582700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe1f00b3cf0 0x7fe1f014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f00630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f00630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56072/0 (socket says 172.21.15.121:56072) 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1f5582700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe1f00b3cf0 0x7fe1f014b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48982/0 (socket says 172.21.15.121:48982) 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 -- 172.21.15.121:0/343553213 learned_addr learned my addr 172.21.15.121:0/343553213 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 -- 172.21.15.121:0/343553213 >> v2:172.21.15.121:3300/0 conn(0x7fe1f00b3cf0 msgr2=0x7fe1f014b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:39.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 --2- 172.21.15.121:0/343553213 >> v2:172.21.15.121:3300/0 conn(0x7fe1f00b3cf0 0x7fe1f014b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 -- 172.21.15.121:0/343553213 >> v1:172.21.15.123:6789/0 conn(0x7fe1f0141440 legacy=0x7fe1f00a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 -- 172.21.15.121:0/343553213 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f00a50b0 con 0x7fe1f009b770 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1f5582700 1 --2- 172.21.15.121:0/343553213 >> v2:172.21.15.121:3300/0 conn(0x7fe1f00b3cf0 0x7fe1f014b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1effff700 1 --2- 172.21.15.121:0/343553213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f00630e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0099b0 tx=0x7fe1dc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=512223196bd58bb1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1ef7fe700 1 -- 172.21.15.121:0/343553213 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc009ea0 con 0x7fe1f009b770 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1ef7fe700 1 -- 172.21.15.121:0/343553213 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc00b860 con 0x7fe1f009b770 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.564+0000 7fe1ef7fe700 1 -- 172.21.15.121:0/343553213 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc00bc00 con 0x7fe1f009b770 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/343553213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 msgr2=0x7fe1f00630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/343553213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f00630e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0099b0 tx=0x7fe1dc009250 comp rx=0 tx=0).stop 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/343553213 shutdown_connections 2023-09-29T23:08:39.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/343553213 >> v2:172.21.15.121:3300/0 conn(0x7fe1f00b3cf0 0x7fe1f014b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/343553213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f00630e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/343553213 >> 172.21.15.121:0/343553213 conn(0x7fe1f0069e90 msgr2=0x7fe1f00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/343553213 shutdown_connections 2023-09-29T23:08:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.565+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/343553213 wait complete. 2023-09-29T23:08:39.568 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 Processor -- start 2023-09-29T23:08:39.571 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 -- start start 2023-09-29T23:08:39.571 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f013b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:39.571 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f00b3cf0 0x7fe1f0142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:39.571 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1f0141440 0x7fe1f0150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:39.571 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0143a80 con 0x7fe1f0141440 2023-09-29T23:08:39.571 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f01512b0 con 0x7fe1f00b3cf0 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.566+0000 7fe1f6fe5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0151570 con 0x7fe1f009b770 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f013b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f5582700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1f0141440 0x7fe1f0150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f5582700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1f0141440 0x7fe1f0150850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48994/0 (socket says 172.21.15.121:48994) 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f013b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56074/0 (socket says 172.21.15.121:56074) 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f5582700 1 -- 172.21.15.121:0/3530357156 learned_addr learned my addr 172.21.15.121:0/3530357156 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1effff700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f00b3cf0 0x7fe1f0142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 -- 172.21.15.121:0/3530357156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f00b3cf0 msgr2=0x7fe1f0142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f00b3cf0 0x7fe1f0142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 -- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1f0141440 msgr2=0x7fe1f0150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:39.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1f0141440 0x7fe1f0150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 -- 172.21.15.121:0/3530357156 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e8009a20 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f5582700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1f0141440 0x7fe1f0150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f4d81700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f013b3a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400c4c0 tx=0x7fe1e4007f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1ecff9700 1 -- 172.21.15.121:0/3530357156 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e400ce30 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1ecff9700 1 -- 172.21.15.121:0/3530357156 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e4011590 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc0094f0 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1ecff9700 1 -- 172.21.15.121:0/3530357156 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4011950 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.567+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f01448c0 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.568+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f0144710 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.569+0000 7fe1ecff9700 1 -- 172.21.15.121:0/3530357156 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe1e4011d40 con 0x7fe1f009b770 2023-09-29T23:08:39.573 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.569+0000 7fe1ecff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.570+0000 7fe1ecff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.570+0000 7fe1ecff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.570+0000 7fe1ecff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.570+0000 7fe1ecff9700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe1d4098780 0x7fe1d409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.570+0000 7fe1effff700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe1d4098780 0x7fe1d409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.570+0000 7fe1ecff9700 1 -- 172.21.15.121:0/3530357156 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fe1e40bfb40 con 0x7fe1f009b770 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.571+0000 7fe1effff700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe1d4098780 0x7fe1d409ac30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe1f009bb60 tx=0x7fe1dc017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:39.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.574+0000 7fe1ecff9700 1 -- 172.21.15.121:0/3530357156 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1e408d160 con 0x7fe1f009b770 2023-09-29T23:08:39.794 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.794+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe1f006ab60 con 0x7fe1f009b770 2023-09-29T23:08:39.794 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.794+0000 7fe1ecff9700 1 -- 172.21.15.121:0/3530357156 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe1e4017020 con 0x7fe1f009b770 2023-09-29T23:08:39.794 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:39.795 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 10 mgrc shutdown 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe1d4098780 msgr2=0x7fe1d409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe1d4098780 0x7fe1d409ac30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe1f009bb60 tx=0x7fe1dc017000 comp rx=0 tx=0).stop 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 msgr2=0x7fe1f013b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f013b3a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400c4c0 tx=0x7fe1e4007f10 comp rx=0 tx=0).stop 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 shutdown_connections 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe1d4098780 0x7fe1d409ac30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.797 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1f0141440 0x7fe1f0150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1f009b770 0x7fe1f013b3a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 --2- 172.21.15.121:0/3530357156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f00b3cf0 0x7fe1f0142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.796+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 >> 172.21.15.121:0/3530357156 conn(0x7fe1f0069e90 msgr2=0x7fe1f013f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.797+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 shutdown_connections 2023-09-29T23:08:39.798 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:39.797+0000 7fe1f6fe5700 1 -- 172.21.15.121:0/3530357156 wait complete. 2023-09-29T23:08:42.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:42.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.938+0000 7f36701ea700 1 Processor -- start 2023-09-29T23:08:42.938 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.938+0000 7f36701ea700 1 -- start start 2023-09-29T23:08:42.940 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f36701ea700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f36680630e0 0x7f366809c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:42.940 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f36701ea700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f366809c700 0x7f36680a1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f36701ea700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f36680b3cf0 0x7f36680a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f36701ea700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36680a0620 con 0x7f36680b3cf0 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f36701ea700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36680a08e0 con 0x7f36680630e0 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f36701ea700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36680a62d0 con 0x7f366809c700 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f366809c700 0x7f36680a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366e787700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f36680b3cf0 0x7f36680a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f366809c700 0x7f36680a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56090/0 (socket says 172.21.15.121:56090) 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 -- 172.21.15.121:0/322401931 learned_addr learned my addr 172.21.15.121:0/322401931 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366df86700 1 --2- 172.21.15.121:0/322401931 >> v2:172.21.15.123:3300/0 conn(0x7f36680630e0 0x7f366809c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:42.941 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 -- 172.21.15.121:0/322401931 >> v2:172.21.15.123:3300/0 conn(0x7f36680630e0 msgr2=0x7f366809c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 --2- 172.21.15.121:0/322401931 >> v2:172.21.15.123:3300/0 conn(0x7f36680630e0 0x7f366809c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 -- 172.21.15.121:0/322401931 >> v2:172.21.15.121:3300/0 conn(0x7f36680b3cf0 msgr2=0x7f36680a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 --2- 172.21.15.121:0/322401931 >> v2:172.21.15.121:3300/0 conn(0x7f36680b3cf0 0x7f36680a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 -- 172.21.15.121:0/322401931 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36680a02c0 con 0x7f366809c700 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366e787700 1 --2- 172.21.15.121:0/322401931 >> v2:172.21.15.121:3300/0 conn(0x7f36680b3cf0 0x7f36680a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.939+0000 7f366d785700 1 --2- 172.21.15.121:0/322401931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f366809c700 0x7f36680a1170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f36680692c0 tx=0x7f3664009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77390e8d28f25ff1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f366cf84700 1 -- 172.21.15.121:0/322401931 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664009e70 con 0x7f366809c700 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f366cf84700 1 -- 172.21.15.121:0/322401931 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366400b860 con 0x7f366809c700 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f366cf84700 1 -- 172.21.15.121:0/322401931 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366400bc00 con 0x7f366809c700 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 -- 172.21.15.121:0/322401931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f366809c700 msgr2=0x7f36680a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 --2- 172.21.15.121:0/322401931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f366809c700 0x7f36680a1170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f36680692c0 tx=0x7f3664009250 comp rx=0 tx=0).stop 2023-09-29T23:08:42.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 -- 172.21.15.121:0/322401931 shutdown_connections 2023-09-29T23:08:42.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 --2- 172.21.15.121:0/322401931 >> v2:172.21.15.121:3300/0 conn(0x7f36680b3cf0 0x7f36680a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:42.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 --2- 172.21.15.121:0/322401931 >> v2:172.21.15.123:3300/0 conn(0x7f36680630e0 0x7f366809c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:42.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 --2- 172.21.15.121:0/322401931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f366809c700 0x7f36680a1170 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:42.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 -- 172.21.15.121:0/322401931 >> 172.21.15.121:0/322401931 conn(0x7f3668069e90 msgr2=0x7f36680b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:42.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 -- 172.21.15.121:0/322401931 shutdown_connections 2023-09-29T23:08:42.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.940+0000 7f36701ea700 1 -- 172.21.15.121:0/322401931 wait complete. 2023-09-29T23:08:42.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.941+0000 7f36701ea700 1 Processor -- start 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.941+0000 7f36701ea700 1 -- start start 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f36701ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36680630e0 0x7f366814ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f36701ea700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f366809c700 0x7f3668099b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f36701ea700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36680b3cf0 0x7f3668098ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f36701ea700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366814e480 con 0x7f36680630e0 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f36701ea700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366814e5e0 con 0x7f366809c700 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f36701ea700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3668098760 con 0x7f36680b3cf0 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36680630e0 0x7f366814ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:42.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36680630e0 0x7f366814ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49022/0 (socket says 172.21.15.121:49022) 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 -- 172.21.15.121:0/3118317220 learned_addr learned my addr 172.21.15.121:0/3118317220 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366e787700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36680b3cf0 0x7f3668098ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366d785700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f366809c700 0x7f3668099b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 -- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36680b3cf0 msgr2=0x7f3668098ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36680b3cf0 0x7f3668098ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 -- 172.21.15.121:0/3118317220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f366809c700 msgr2=0x7f3668099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f366809c700 0x7f3668099b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 -- 172.21.15.121:0/3118317220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3660009a20 con 0x7f36680630e0 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.942+0000 7f366df86700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36680630e0 0x7f366814ea70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f365800d810 tx=0x7f3658008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.943+0000 7f365dffb700 1 -- 172.21.15.121:0/3118317220 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36580125f0 con 0x7f36680630e0 2023-09-29T23:08:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.943+0000 7f365dffb700 1 -- 172.21.15.121:0/3118317220 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658012750 con 0x7f36680630e0 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.943+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36640094f0 con 0x7f36680630e0 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.943+0000 7f365dffb700 1 -- 172.21.15.121:0/3118317220 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658012b10 con 0x7f36680630e0 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.943+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366806f480 con 0x7f36680630e0 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.944+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366814e8c0 con 0x7f36680630e0 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f365dffb700 1 -- 172.21.15.121:0/3118317220 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f365801e460 con 0x7f36680630e0 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f365dffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f365dffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f365dffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f365dffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f365dffb700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f364c098700 0x7f364c09abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f366d785700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f364c098700 0x7f364c09abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:42.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.945+0000 7f365dffb700 1 -- 172.21.15.121:0/3118317220 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f36580c0ac0 con 0x7f36680630e0 2023-09-29T23:08:42.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.946+0000 7f366d785700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f364c098700 0x7f364c09abb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36680692c0 tx=0x7f3664017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:42.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:42.949+0000 7f365dffb700 1 -- 172.21.15.121:0/3118317220 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f365808e170 con 0x7f36680630e0 2023-09-29T23:08:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.171+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f366806a260 con 0x7f36680630e0 2023-09-29T23:08:43.171 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.171+0000 7f365dffb700 1 -- 172.21.15.121:0/3118317220 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f36580c0e60 con 0x7f36680630e0 2023-09-29T23:08:43.171 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:43.171 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:43.173 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 10 mgrc shutdown 2023-09-29T23:08:43.173 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f364c098700 msgr2=0x7f364c09abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:43.173 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f364c098700 0x7f364c09abb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36680692c0 tx=0x7f3664017000 comp rx=0 tx=0).stop 2023-09-29T23:08:43.173 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36680630e0 msgr2=0x7f366814ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:43.173 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36680630e0 0x7f366814ea70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f365800d810 tx=0x7f3658008040 comp rx=0 tx=0).stop 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 shutdown_connections 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f364c098700 0x7f364c09abb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36680b3cf0 0x7f3668098ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36680630e0 0x7f366814ea70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 --2- 172.21.15.121:0/3118317220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f366809c700 0x7f3668099b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 >> 172.21.15.121:0/3118317220 conn(0x7f3668069e90 msgr2=0x7f36680b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 shutdown_connections 2023-09-29T23:08:43.174 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:43.173+0000 7f36701ea700 1 -- 172.21.15.121:0/3118317220 wait complete. 2023-09-29T23:08:46.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:46.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.319+0000 7fb63b3ca700 1 Processor -- start 2023-09-29T23:08:46.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.319+0000 7fb63b3ca700 1 -- start start 2023-09-29T23:08:46.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.319+0000 7fb63b3ca700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb6340a25d0 0x7fb634140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:46.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb63b3ca700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6340a5800 0x7fb634140da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:46.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb63b3ca700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb634068fc0 con 0x7fb634068bf0 2023-09-29T23:08:46.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb63b3ca700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340a1070 con 0x7fb6340a5800 2023-09-29T23:08:46.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb63b3ca700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6341412e0 con 0x7fb6340a25d0 2023-09-29T23:08:46.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6340a5800 0x7fb634140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:46.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6340a5800 0x7fb634140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38488/0 (socket says 172.21.15.121:38488) 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 -- 172.21.15.121:0/205004162 learned_addr learned my addr 172.21.15.121:0/205004162 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb638965700 1 --2- 172.21.15.121:0/205004162 >> v2:172.21.15.123:3300/0 conn(0x7fb6340a25d0 0x7fb634140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 -- 172.21.15.121:0/205004162 >> v2:172.21.15.123:3300/0 conn(0x7fb6340a25d0 msgr2=0x7fb634140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 --2- 172.21.15.121:0/205004162 >> v2:172.21.15.123:3300/0 conn(0x7fb6340a25d0 0x7fb634140860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 -- 172.21.15.121:0/205004162 >> v1:172.21.15.123:6789/0 conn(0x7fb634068bf0 legacy=0x7fb63413ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 -- 172.21.15.121:0/205004162 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6341414f0 con 0x7fb6340a5800 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb639967700 1 --2- 172.21.15.121:0/205004162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a5800 0x7fb634140da0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb63406ace0 tx=0x7fb62c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405840bf42a5bccf server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb62bfff700 1 -- 172.21.15.121:0/205004162 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00e070 con 0x7fb6340a5800 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb62bfff700 1 -- 172.21.15.121:0/205004162 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c00ba10 con 0x7fb6340a5800 2023-09-29T23:08:46.322 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.320+0000 7fb62bfff700 1 -- 172.21.15.121:0/205004162 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0092e0 con 0x7fb6340a5800 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 -- 172.21.15.121:0/205004162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a5800 msgr2=0x7fb634140da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/205004162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a5800 0x7fb634140da0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb63406ace0 tx=0x7fb62c009840 comp rx=0 tx=0).stop 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 -- 172.21.15.121:0/205004162 shutdown_connections 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/205004162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a5800 0x7fb634140da0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/205004162 >> v2:172.21.15.123:3300/0 conn(0x7fb6340a25d0 0x7fb634140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 -- 172.21.15.121:0/205004162 >> 172.21.15.121:0/205004162 conn(0x7fb634067440 msgr2=0x7fb634067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 -- 172.21.15.121:0/205004162 shutdown_connections 2023-09-29T23:08:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.321+0000 7fb63b3ca700 1 -- 172.21.15.121:0/205004162 wait complete. 2023-09-29T23:08:46.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.322+0000 7fb63b3ca700 1 Processor -- start 2023-09-29T23:08:46.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.322+0000 7fb63b3ca700 1 -- start start 2023-09-29T23:08:46.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb63b3ca700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb634068bf0 0x7fb6341402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:46.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb63b3ca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a25d0 0x7fb634135f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:46.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb63b3ca700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6340a5800 0x7fb63406e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:46.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb63b3ca700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb634140800 con 0x7fb6340a25d0 2023-09-29T23:08:46.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb63b3ca700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63406ea60 con 0x7fb6340a5800 2023-09-29T23:08:46.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb63b3ca700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb634137f90 con 0x7fb634068bf0 2023-09-29T23:08:46.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb634068bf0 0x7fb6341402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:46.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb638965700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a25d0 0x7fb634135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:46.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb634068bf0 0x7fb6341402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:42620/0 (socket says 172.21.15.121:42620) 2023-09-29T23:08:46.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 -- 172.21.15.121:0/694237581 learned_addr learned my addr 172.21.15.121:0/694237581 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:46.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639967700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6340a5800 0x7fb63406e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:46.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 -- 172.21.15.121:0/694237581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6340a5800 msgr2=0x7fb63406e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:46.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6340a5800 0x7fb63406e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 -- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a25d0 msgr2=0x7fb634135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:46.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a25d0 0x7fb634135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 -- 172.21.15.121:0/694237581 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c009ae0 con 0x7fb634068bf0 2023-09-29T23:08:46.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb638965700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a25d0 0x7fb634135f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:46.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb639166700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb634068bf0 0x7fb6341402c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb63000c5d0 tx=0x7fb630007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:46.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb6297fa700 1 -- 172.21.15.121:0/694237581 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630011590 con 0x7fb634068bf0 2023-09-29T23:08:46.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb6297fa700 1 -- 172.21.15.121:0/694237581 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb630011710 con 0x7fb634068bf0 2023-09-29T23:08:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.323+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb634138250 con 0x7fb634068bf0 2023-09-29T23:08:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.324+0000 7fb6297fa700 1 -- 172.21.15.121:0/694237581 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630011ad0 con 0x7fb634068bf0 2023-09-29T23:08:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.324+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6341364b0 con 0x7fb634068bf0 2023-09-29T23:08:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.325+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb634071660 con 0x7fb634068bf0 2023-09-29T23:08:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.325+0000 7fb6297fa700 1 -- 172.21.15.121:0/694237581 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb63001c460 con 0x7fb634068bf0 2023-09-29T23:08:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.325+0000 7fb6297fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:46.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.326+0000 7fb6297fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:46.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.326+0000 7fb6297fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:46.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.326+0000 7fb6297fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:46.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.326+0000 7fb6297fa700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb618098780 0x7fb61809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:46.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.326+0000 7fb638965700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb618098780 0x7fb61809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:46.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.326+0000 7fb6297fa700 1 -- 172.21.15.121:0/694237581 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fb6300c0220 con 0x7fb634068bf0 2023-09-29T23:08:46.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.326+0000 7fb638965700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb618098780 0x7fb61809ac30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb6341335f0 tx=0x7fb624008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:46.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.330+0000 7fb6297fa700 1 -- 172.21.15.121:0/694237581 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb63008d840 con 0x7fb634068bf0 2023-09-29T23:08:46.553 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.553+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb63413d160 con 0x7fb634068bf0 2023-09-29T23:08:46.553 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.553+0000 7fb6297fa700 1 -- 172.21.15.121:0/694237581 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb630090e60 con 0x7fb634068bf0 2023-09-29T23:08:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:46.553 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:46.556 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.555+0000 7fb63b3ca700 10 mgrc shutdown 2023-09-29T23:08:46.556 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.555+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb618098780 msgr2=0x7fb61809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:46.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.555+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb618098780 0x7fb61809ac30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb6341335f0 tx=0x7fb624008040 comp rx=0 tx=0).stop 2023-09-29T23:08:46.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.555+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb634068bf0 msgr2=0x7fb6341402c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:46.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.555+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb634068bf0 0x7fb6341402c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb63000c5d0 tx=0x7fb630007f60 comp rx=0 tx=0).stop 2023-09-29T23:08:46.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 shutdown_connections 2023-09-29T23:08:46.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb618098780 0x7fb61809ac30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6340a5800 0x7fb63406e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb634068bf0 0x7fb6341402c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 --2- 172.21.15.121:0/694237581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6340a25d0 0x7fb634135f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:46.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 >> 172.21.15.121:0/694237581 conn(0x7fb634067440 msgr2=0x7fb6340b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:46.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 shutdown_connections 2023-09-29T23:08:46.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:46.556+0000 7fb63b3ca700 1 -- 172.21.15.121:0/694237581 wait complete. 2023-09-29T23:08:49.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.699+0000 7f74726bd700 1 Processor -- start 2023-09-29T23:08:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f74726bd700 1 -- start start 2023-09-29T23:08:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f74726bd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f746c141440 0x7f746c0a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f74726bd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f746c0b3cf0 0x7f746c14b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f74726bd700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746c141810 con 0x7f746c09b770 2023-09-29T23:08:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f74726bd700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f746c09a7f0 con 0x7f746c141440 2023-09-29T23:08:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f74726bd700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f746c0a4df0 con 0x7f746c0b3cf0 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f746bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f746c141440 0x7f746c0a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f746bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f746c141440 0x7f746c0a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38498/0 (socket says 172.21.15.121:38498) 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f746b7fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f746c09b770 0x7f746c0630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47910/0 (socket says 172.21.15.121:47910) 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.700+0000 7f746bfff700 1 -- 172.21.15.121:0/454923077 learned_addr learned my addr 172.21.15.121:0/454923077 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f7470c5a700 1 --2- 172.21.15.121:0/454923077 >> v2:172.21.15.123:3300/0 conn(0x7f746c0b3cf0 0x7f746c14b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746bfff700 1 -- 172.21.15.121:0/454923077 >> v2:172.21.15.123:3300/0 conn(0x7f746c0b3cf0 msgr2=0x7f746c14b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746bfff700 1 --2- 172.21.15.121:0/454923077 >> v2:172.21.15.123:3300/0 conn(0x7f746c0b3cf0 0x7f746c14b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746bfff700 1 -- 172.21.15.121:0/454923077 >> v1:172.21.15.121:6789/0 conn(0x7f746c09b770 legacy=0x7f746c0630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746bfff700 1 -- 172.21.15.121:0/454923077 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c0a50b0 con 0x7f746c141440 2023-09-29T23:08:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746affd700 1 -- 172.21.15.121:0/454923077 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181716480 0 0) 0x7f746c141810 con 0x7f746c09b770 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746bfff700 1 --2- 172.21.15.121:0/454923077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c141440 0x7f746c0a45a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f746c06b4e0 tx=0x7f745c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa4dd5270ec91c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746affd700 1 -- 172.21.15.121:0/454923077 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c005c10 con 0x7f746c141440 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746affd700 1 -- 172.21.15.121:0/454923077 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745c005d70 con 0x7f746c141440 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f746affd700 1 -- 172.21.15.121:0/454923077 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c00b8a0 con 0x7f746c141440 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 -- 172.21.15.121:0/454923077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c141440 msgr2=0x7f746c0a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 --2- 172.21.15.121:0/454923077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c141440 0x7f746c0a45a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f746c06b4e0 tx=0x7f745c009220 comp rx=0 tx=0).stop 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 -- 172.21.15.121:0/454923077 shutdown_connections 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 --2- 172.21.15.121:0/454923077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c141440 0x7f746c0a45a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 --2- 172.21.15.121:0/454923077 >> v2:172.21.15.123:3300/0 conn(0x7f746c0b3cf0 0x7f746c14b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 -- 172.21.15.121:0/454923077 >> 172.21.15.121:0/454923077 conn(0x7f746c069e90 msgr2=0x7f746c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 -- 172.21.15.121:0/454923077 shutdown_connections 2023-09-29T23:08:49.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.701+0000 7f74726bd700 1 -- 172.21.15.121:0/454923077 wait complete. 2023-09-29T23:08:49.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.702+0000 7f74726bd700 1 Processor -- start 2023-09-29T23:08:49.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f74726bd700 1 -- start start 2023-09-29T23:08:49.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f74726bd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f746c09b770 0x7f746c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:49.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f74726bd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c0b3cf0 0x7f746c1538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:49.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f74726bd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f746c141440 0x7f746c143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f74726bd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c1541f0 con 0x7f746c0b3cf0 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f74726bd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c0a2580 con 0x7f746c09b770 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f74726bd700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746c144ff0 con 0x7f746c141440 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f746b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c0b3cf0 0x7f746c1538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f7470c5a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f746c141440 0x7f746c143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f746b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c0b3cf0 0x7f746c1538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38514/0 (socket says 172.21.15.121:38514) 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f746b7fe700 1 -- 172.21.15.121:0/2894102989 learned_addr learned my addr 172.21.15.121:0/2894102989 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.703+0000 7f746bfff700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f746c09b770 0x7f746c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f746b7fe700 1 -- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f746c141440 msgr2=0x7f746c143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f746b7fe700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f746c141440 0x7f746c143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f746b7fe700 1 -- 172.21.15.121:0/2894102989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f746c09b770 msgr2=0x7f746c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f746b7fe700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f746c09b770 0x7f746c06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f746b7fe700 1 -- 172.21.15.121:0/2894102989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c009480 con 0x7f746c0b3cf0 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f746bfff700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f746c09b770 0x7f746c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f746b7fe700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c0b3cf0 0x7f746c1538b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f746000c7e0 tx=0x7f7460007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f7453fff700 1 -- 172.21.15.121:0/2894102989 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460005b40 con 0x7f746c0b3cf0 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f7453fff700 1 -- 172.21.15.121:0/2894102989 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7460005ca0 con 0x7f746c0b3cf0 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746c1452b0 con 0x7f746c0b3cf0 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f7453fff700 1 -- 172.21.15.121:0/2894102989 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460007320 con 0x7f746c0b3cf0 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.704+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746c1546c0 con 0x7f746c0b3cf0 2023-09-29T23:08:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.705+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746c154360 con 0x7f746c0b3cf0 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f7453fff700 1 -- 172.21.15.121:0/2894102989 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f746001b3b0 con 0x7f746c0b3cf0 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f7453fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f7453fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f7453fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f7453fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f7453fff700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7458098780 0x7f745809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f746bfff700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7458098780 0x7f745809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.706+0000 7f7453fff700 1 -- 172.21.15.121:0/2894102989 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f74600bf9a0 con 0x7f746c0b3cf0 2023-09-29T23:08:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.707+0000 7f746bfff700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7458098780 0x7f745809ac30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f746c06b4e0 tx=0x7f745c016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.710+0000 7f7453fff700 1 -- 172.21.15.121:0/2894102989 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f746c154360 con 0x7f746c0b3cf0 2023-09-29T23:08:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.930+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f746c06ab60 con 0x7f746c0b3cf0 2023-09-29T23:08:49.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.931+0000 7f7453fff700 1 -- 172.21.15.121:0/2894102989 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7460017100 con 0x7f746c0b3cf0 2023-09-29T23:08:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:49.931 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 10 mgrc shutdown 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7458098780 msgr2=0x7f745809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7458098780 0x7f745809ac30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f746c06b4e0 tx=0x7f745c016000 comp rx=0 tx=0).stop 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c0b3cf0 msgr2=0x7f746c1538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c0b3cf0 0x7f746c1538b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f746000c7e0 tx=0x7f7460007de0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 shutdown_connections 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7458098780 0x7f745809ac30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f746c141440 0x7f746c143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.934 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f746c09b770 0x7f746c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.934 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 --2- 172.21.15.121:0/2894102989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f746c0b3cf0 0x7f746c1538b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:49.934 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 >> 172.21.15.121:0/2894102989 conn(0x7f746c069e90 msgr2=0x7f746c0a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:49.934 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 shutdown_connections 2023-09-29T23:08:49.934 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:49.933+0000 7f74726bd700 1 -- 172.21.15.121:0/2894102989 wait complete. 2023-09-29T23:08:52.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.074+0000 7fe060e8c700 1 Processor -- start 2023-09-29T23:08:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.074+0000 7fe060e8c700 1 -- start start 2023-09-29T23:08:53.076 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe060e8c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c09c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:53.076 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe060e8c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe05c09c700 0x7fe05c0a1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe060e8c700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c0a0620 con 0x7fe05c0b3cf0 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe060e8c700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05c0a08e0 con 0x7fe05c09c700 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe060e8c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0a62d0 con 0x7fe05c0630e0 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe059d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe05c09c700 0x7fe05c0a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe05a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c09c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe059d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe05c09c700 0x7fe05c0a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38528/0 (socket says 172.21.15.121:38528) 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe05a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c09c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:42632/0 (socket says 172.21.15.121:42632) 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe05ad9d700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe05c0b3cf0 0x7fe05c0a16b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47914/0 (socket says 172.21.15.121:47914) 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.075+0000 7fe059d9b700 1 -- 172.21.15.121:0/1466931817 learned_addr learned my addr 172.21.15.121:0/1466931817 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05a59c700 1 -- 172.21.15.121:0/1466931817 >> v2:172.21.15.121:3300/0 conn(0x7fe05c09c700 msgr2=0x7fe05c0a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:53.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05a59c700 1 --2- 172.21.15.121:0/1466931817 >> v2:172.21.15.121:3300/0 conn(0x7fe05c09c700 0x7fe05c0a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05a59c700 1 -- 172.21.15.121:0/1466931817 >> v1:172.21.15.121:6789/0 conn(0x7fe05c0b3cf0 legacy=0x7fe05c0a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05a59c700 1 -- 172.21.15.121:0/1466931817 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0a02c0 con 0x7fe05c0630e0 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05959a700 1 -- 172.21.15.121:0/1466931817 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230745625 0 0) 0x7fe05c0a0620 con 0x7fe05c0b3cf0 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe059d9b700 1 --2- 172.21.15.121:0/1466931817 >> v2:172.21.15.121:3300/0 conn(0x7fe05c09c700 0x7fe05c0a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05a59c700 1 --2- 172.21.15.121:0/1466931817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c09c1c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe0500098d0 tx=0x7fe050009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b66becebf57d52 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05959a700 1 -- 172.21.15.121:0/1466931817 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050005c10 con 0x7fe05c0630e0 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05959a700 1 -- 172.21.15.121:0/1466931817 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe050005d70 con 0x7fe05c0630e0 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.076+0000 7fe05959a700 1 -- 172.21.15.121:0/1466931817 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050006280 con 0x7fe05c0630e0 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 -- 172.21.15.121:0/1466931817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 msgr2=0x7fe05c09c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 --2- 172.21.15.121:0/1466931817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c09c1c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe0500098d0 tx=0x7fe050009120 comp rx=0 tx=0).stop 2023-09-29T23:08:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 -- 172.21.15.121:0/1466931817 shutdown_connections 2023-09-29T23:08:53.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 --2- 172.21.15.121:0/1466931817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c09c1c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 --2- 172.21.15.121:0/1466931817 >> v2:172.21.15.121:3300/0 conn(0x7fe05c09c700 0x7fe05c0a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 -- 172.21.15.121:0/1466931817 >> 172.21.15.121:0/1466931817 conn(0x7fe05c069e90 msgr2=0x7fe05c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:53.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 -- 172.21.15.121:0/1466931817 shutdown_connections 2023-09-29T23:08:53.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.077+0000 7fe060e8c700 1 -- 172.21.15.121:0/1466931817 wait complete. 2023-09-29T23:08:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 Processor -- start 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 -- start start 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c06e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe05c09c700 0x7fe05c095a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe05c0b3cf0 0x7fe05c095f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c096960 con 0x7fe05c0b3cf0 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c096490 con 0x7fe05c09c700 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe060e8c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0966a0 con 0x7fe05c0630e0 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe05a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c06e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe05a59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c06e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:42646/0 (socket says 172.21.15.121:42646) 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe05a59c700 1 -- 172.21.15.121:0/2483269463 learned_addr learned my addr 172.21.15.121:0/2483269463 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.078+0000 7fe05ad9d700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe05c0b3cf0 0x7fe05c095f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe059d9b700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe05c09c700 0x7fe05c095a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe05a59c700 1 -- 172.21.15.121:0/2483269463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe05c09c700 msgr2=0x7fe05c095a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe05a59c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe05c09c700 0x7fe05c095a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe05a59c700 1 -- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe05c0b3cf0 msgr2=0x7fe05c095f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe05a59c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe05c0b3cf0 0x7fe05c095f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe05a59c700 1 -- 172.21.15.121:0/2483269463 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c0094f0 con 0x7fe05c0630e0 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe05a59c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c06e9e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe050009f70 tx=0x7fe050016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe04a7fc700 1 -- 172.21.15.121:0/2483269463 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05001d780 con 0x7fe05c0630e0 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe04a7fc700 1 -- 172.21.15.121:0/2483269463 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05001d8e0 con 0x7fe05c0630e0 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0500093c0 con 0x7fe05c0630e0 2023-09-29T23:08:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe04a7fc700 1 -- 172.21.15.121:0/2483269463 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05001dca0 con 0x7fe05c0630e0 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.079+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c06f9d0 con 0x7fe05c0630e0 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.080+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c06ab60 con 0x7fe05c0630e0 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.081+0000 7fe04a7fc700 1 -- 172.21.15.121:0/2483269463 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe05001b690 con 0x7fe05c0630e0 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.081+0000 7fe04a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.081+0000 7fe04a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.081+0000 7fe04a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.081+0000 7fe04a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.081+0000 7fe04a7fc700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe03c098780 0x7fe03c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.081+0000 7fe059d9b700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe03c098780 0x7fe03c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.082+0000 7fe04a7fc700 1 -- 172.21.15.121:0/2483269463 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fe0500c2440 con 0x7fe05c0630e0 2023-09-29T23:08:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.082+0000 7fe059d9b700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe03c098780 0x7fe03c09ac30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe05c097300 tx=0x7fe04c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:53.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.086+0000 7fe04a7fc700 1 -- 172.21.15.121:0/2483269463 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe05008faf0 con 0x7fe05c0630e0 2023-09-29T23:08:53.305 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.306+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe05c071660 con 0x7fe05c0630e0 2023-09-29T23:08:53.306 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.306+0000 7fe04a7fc700 1 -- 172.21.15.121:0/2483269463 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe050093110 con 0x7fe05c0630e0 2023-09-29T23:08:53.306 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:53.306 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:53.308 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 10 mgrc shutdown 2023-09-29T23:08:53.308 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe03c098780 msgr2=0x7fe03c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:53.308 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe03c098780 0x7fe03c09ac30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe05c097300 tx=0x7fe04c008040 comp rx=0 tx=0).stop 2023-09-29T23:08:53.308 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 msgr2=0x7fe05c06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:53.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c06e9e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe050009f70 tx=0x7fe050016000 comp rx=0 tx=0).stop 2023-09-29T23:08:53.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 shutdown_connections 2023-09-29T23:08:53.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe03c098780 0x7fe03c09ac30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe05c0b3cf0 0x7fe05c095f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe05c0630e0 0x7fe05c06e9e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.309 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 --2- 172.21.15.121:0/2483269463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe05c09c700 0x7fe05c095a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:53.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 >> 172.21.15.121:0/2483269463 conn(0x7fe05c069e90 msgr2=0x7fe05c0b2610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:53.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 shutdown_connections 2023-09-29T23:08:53.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:53.308+0000 7fe060e8c700 1 -- 172.21.15.121:0/2483269463 wait complete. 2023-09-29T23:08:56.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:56.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.450+0000 7f5807097700 1 Processor -- start 2023-09-29T23:08:56.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.450+0000 7f5807097700 1 -- start start 2023-09-29T23:08:56.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.450+0000 7f5807097700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:56.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.450+0000 7f5807097700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f58000b3cf0 0x7f580014b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:56.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.450+0000 7f5807097700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5800141810 con 0x7f5800141440 2023-09-29T23:08:56.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.450+0000 7f5807097700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580009a7f0 con 0x7f58000b3cf0 2023-09-29T23:08:56.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.450+0000 7f5807097700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58000a4df0 con 0x7f580009b770 2023-09-29T23:08:56.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57fffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:56.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f5804e33700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5800141440 0x7f58000a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:36504/0 (socket says 172.21.15.121:36504) 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f5804e33700 1 -- 172.21.15.121:0/242027543 learned_addr learned my addr 172.21.15.121:0/242027543 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57fffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54090/0 (socket says 172.21.15.121:54090) 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f5805634700 1 --2- 172.21.15.121:0/242027543 >> v2:172.21.15.123:3300/0 conn(0x7f58000b3cf0 0x7f580014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57fffff700 1 -- 172.21.15.121:0/242027543 >> v2:172.21.15.123:3300/0 conn(0x7f58000b3cf0 msgr2=0x7f580014b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57fffff700 1 --2- 172.21.15.121:0/242027543 >> v2:172.21.15.123:3300/0 conn(0x7f58000b3cf0 0x7f580014b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57fffff700 1 -- 172.21.15.121:0/242027543 >> v1:172.21.15.121:6789/0 conn(0x7f5800141440 legacy=0x7f58000a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57ff7fe700 1 -- 172.21.15.121:0/242027543 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116890576 0 0) 0x7f5800141810 con 0x7f5800141440 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57fffff700 1 -- 172.21.15.121:0/242027543 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58000a50b0 con 0x7f580009b770 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57fffff700 1 --2- 172.21.15.121:0/242027543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000630e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0098e0 tx=0x7f57ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccead58367276532 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:56.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57ff7fe700 1 -- 172.21.15.121:0/242027543 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec009e80 con 0x7f580009b770 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57ff7fe700 1 -- 172.21.15.121:0/242027543 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec00b860 con 0x7f580009b770 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.451+0000 7f57ff7fe700 1 -- 172.21.15.121:0/242027543 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec00bbe0 con 0x7f580009b770 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 -- 172.21.15.121:0/242027543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 msgr2=0x7f58000630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 --2- 172.21.15.121:0/242027543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000630e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0098e0 tx=0x7f57ec0091a0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 -- 172.21.15.121:0/242027543 shutdown_connections 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 --2- 172.21.15.121:0/242027543 >> v2:172.21.15.123:3300/0 conn(0x7f58000b3cf0 0x7f580014b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 --2- 172.21.15.121:0/242027543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000630e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 -- 172.21.15.121:0/242027543 >> 172.21.15.121:0/242027543 conn(0x7f5800069e90 msgr2=0x7f58000b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 -- 172.21.15.121:0/242027543 shutdown_connections 2023-09-29T23:08:56.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.452+0000 7f5807097700 1 -- 172.21.15.121:0/242027543 wait complete. 2023-09-29T23:08:56.457 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.453+0000 7f5807097700 1 Processor -- start 2023-09-29T23:08:56.457 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.453+0000 7f5807097700 1 -- start start 2023-09-29T23:08:56.457 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.453+0000 7f5807097700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.453+0000 7f5807097700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58000b3cf0 0x7f580013acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.453+0000 7f5807097700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5800141440 0x7f580013b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5807097700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000a1510 con 0x7f58000b3cf0 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5807097700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000a24a0 con 0x7f5800141440 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5807097700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58001543b0 con 0x7f580009b770 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f57fffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58000b3cf0 0x7f580013acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5804e33700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f57fffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58000b3cf0 0x7f580013acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50838/0 (socket says 172.21.15.121:50838) 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f57fffff700 1 -- 172.21.15.121:0/3834800369 learned_addr learned my addr 172.21.15.121:0/3834800369 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:56.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5805634700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5800141440 0x7f580013b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5804e33700 1 -- 172.21.15.121:0/3834800369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5800141440 msgr2=0x7f580013b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5804e33700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5800141440 0x7f580013b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5804e33700 1 -- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58000b3cf0 msgr2=0x7f580013acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5804e33700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58000b3cf0 0x7f580013acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5804e33700 1 -- 172.21.15.121:0/3834800369 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec009420 con 0x7f580009b770 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f57fffff700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58000b3cf0 0x7f580013acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5804e33700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000a1f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f57f400c4e0 tx=0x7f57f4007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f57fcff9700 1 -- 172.21.15.121:0/3834800369 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f40073b0 con 0x7f580009b770 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f57fcff9700 1 -- 172.21.15.121:0/3834800369 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f4007510 con 0x7f580009b770 2023-09-29T23:08:56.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5800154670 con 0x7f580009b770 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f57fcff9700 1 -- 172.21.15.121:0/3834800369 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4005a40 con 0x7f580009b770 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.454+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58000a17f0 con 0x7f580009b770 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.455+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580006ab60 con 0x7f580009b770 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.456+0000 7f57fcff9700 1 -- 172.21.15.121:0/3834800369 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f57f401b3b0 con 0x7f580009b770 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.456+0000 7f57fcff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.456+0000 7f57fcff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.456+0000 7f57fcff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.456+0000 7f57fcff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.456+0000 7f57fcff9700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f57e4098780 0x7f57e409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.457+0000 7f57fffff700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f57e4098780 0x7f57e409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:56.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.457+0000 7f57fcff9700 1 -- 172.21.15.121:0/3834800369 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f57f40bf9e0 con 0x7f580009b770 2023-09-29T23:08:56.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.457+0000 7f57fffff700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f57e4098780 0x7f57e409ac30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58001515e0 tx=0x7f57ec017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:56.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.461+0000 7f57fcff9700 1 -- 172.21.15.121:0/3834800369 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57f408d000 con 0x7f580009b770 2023-09-29T23:08:56.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.681+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5800071660 con 0x7f580009b770 2023-09-29T23:08:56.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.681+0000 7f57fcff9700 1 -- 172.21.15.121:0/3834800369 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f57f4017080 con 0x7f580009b770 2023-09-29T23:08:56.681 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:08:56.681 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:08:56.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.683+0000 7f5807097700 10 mgrc shutdown 2023-09-29T23:08:56.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.683+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f57e4098780 msgr2=0x7f57e409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:56.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.683+0000 7f5807097700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f57e4098780 0x7f57e409ac30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58001515e0 tx=0x7f57ec017000 comp rx=0 tx=0).stop 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.683+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 msgr2=0x7f58000a1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.683+0000 7f5807097700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000a1f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f57f400c4e0 tx=0x7f57f4007dc0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 shutdown_connections 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f57e4098780 0x7f57e409ac30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5800141440 0x7f580013b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f580009b770 0x7f58000a1f60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 --2- 172.21.15.121:0/3834800369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58000b3cf0 0x7f580013acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 >> 172.21.15.121:0/3834800369 conn(0x7f5800069e90 msgr2=0x7f580013c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 shutdown_connections 2023-09-29T23:08:56.685 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:56.684+0000 7f5807097700 1 -- 172.21.15.121:0/3834800369 wait complete. 2023-09-29T23:08:59.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:08:59.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.824+0000 7f6f1298a700 1 Processor -- start 2023-09-29T23:08:59.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f1298a700 1 -- start start 2023-09-29T23:08:59.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f1298a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6f0c0630e0 0x7f6f0c09c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:59.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f1298a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f0c09f580 0x7f6f0c09ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:59.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f1298a700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c09e600 con 0x7f6f0c0634b0 2023-09-29T23:08:59.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f1298a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0c0a2d20 con 0x7f6f0c0630e0 2023-09-29T23:08:59.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f1298a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0c09d8d0 con 0x7f6f0c09f580 2023-09-29T23:08:59.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f0bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6f0c0630e0 0x7f6f0c09c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f0bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6f0c0630e0 0x7f6f0c09c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50842/0 (socket says 172.21.15.121:50842) 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f0b7fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6f0c0634b0 0x7f6f0c09c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:36512/0 (socket says 172.21.15.121:36512) 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.825+0000 7f6f0bfff700 1 -- 172.21.15.121:0/3330332691 learned_addr learned my addr 172.21.15.121:0/3330332691 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f10f27700 1 --2- 172.21.15.121:0/3330332691 >> v2:172.21.15.123:3300/0 conn(0x7f6f0c09f580 0x7f6f0c09ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0bfff700 1 -- 172.21.15.121:0/3330332691 >> v2:172.21.15.123:3300/0 conn(0x7f6f0c09f580 msgr2=0x7f6f0c09ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0bfff700 1 --2- 172.21.15.121:0/3330332691 >> v2:172.21.15.123:3300/0 conn(0x7f6f0c09f580 0x7f6f0c09ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0bfff700 1 -- 172.21.15.121:0/3330332691 >> v1:172.21.15.121:6789/0 conn(0x7f6f0c0634b0 legacy=0x7f6f0c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0bfff700 1 -- 172.21.15.121:0/3330332691 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c09db70 con 0x7f6f0c0630e0 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f10f27700 1 --2- 172.21.15.121:0/3330332691 >> v2:172.21.15.123:3300/0 conn(0x7f6f0c09f580 0x7f6f0c09ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0bfff700 1 --2- 172.21.15.121:0/3330332691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0630e0 0x7f6f0c09c070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c06bb40 tx=0x7f6efc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb1041ed4486547 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0affd700 1 -- 172.21.15.121:0/3330332691 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc005c10 con 0x7f6f0c0630e0 2023-09-29T23:08:59.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0affd700 1 -- 172.21.15.121:0/3330332691 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc005d70 con 0x7f6f0c0630e0 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f0affd700 1 -- 172.21.15.121:0/3330332691 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00b880 con 0x7f6f0c0630e0 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f1298a700 1 -- 172.21.15.121:0/3330332691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0630e0 msgr2=0x7f6f0c09c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f1298a700 1 --2- 172.21.15.121:0/3330332691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0630e0 0x7f6f0c09c070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c06bb40 tx=0x7f6efc009220 comp rx=0 tx=0).stop 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f1298a700 1 -- 172.21.15.121:0/3330332691 shutdown_connections 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f1298a700 1 --2- 172.21.15.121:0/3330332691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0630e0 0x7f6f0c09c070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f1298a700 1 --2- 172.21.15.121:0/3330332691 >> v2:172.21.15.123:3300/0 conn(0x7f6f0c09f580 0x7f6f0c09ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f1298a700 1 -- 172.21.15.121:0/3330332691 >> 172.21.15.121:0/3330332691 conn(0x7f6f0c069e90 msgr2=0x7f6f0c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.826+0000 7f6f1298a700 1 -- 172.21.15.121:0/3330332691 shutdown_connections 2023-09-29T23:08:59.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.827+0000 7f6f1298a700 1 -- 172.21.15.121:0/3330332691 wait complete. 2023-09-29T23:08:59.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.827+0000 7f6f1298a700 1 Processor -- start 2023-09-29T23:08:59.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f1298a700 1 -- start start 2023-09-29T23:08:59.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f1298a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f0c0630e0 0x7f6f0c095d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:59.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f1298a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0634b0 0x7f6f0c14bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:59.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f1298a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c09f580 0x7f6f0c097420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:59.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f1298a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c096f90 con 0x7f6f0c0634b0 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f1298a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c06dc50 con 0x7f6f0c0630e0 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f1298a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c096950 con 0x7f6f0c09f580 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f0b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0634b0 0x7f6f0c14bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f10f27700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c09f580 0x7f6f0c097420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f0b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0634b0 0x7f6f0c14bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50844/0 (socket says 172.21.15.121:50844) 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f0b7fe700 1 -- 172.21.15.121:0/1477572896 learned_addr learned my addr 172.21.15.121:0/1477572896 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.828+0000 7f6f0bfff700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f0c0630e0 0x7f6f0c095d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f0b7fe700 1 -- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c09f580 msgr2=0x7f6f0c097420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f0b7fe700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c09f580 0x7f6f0c097420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f0b7fe700 1 -- 172.21.15.121:0/1477572896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f0c0630e0 msgr2=0x7f6f0c095d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:08:59.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f0b7fe700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f0c0630e0 0x7f6f0c095d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f0b7fe700 1 -- 172.21.15.121:0/1477572896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc009480 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f0b7fe700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0634b0 0x7f6f0c14bde0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c7c0 tx=0x7f6f00007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6ef3fff700 1 -- 172.21.15.121:0/1477572896 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00005a70 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6ef3fff700 1 -- 172.21.15.121:0/1477572896 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00005bd0 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c14c4a0 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6ef3fff700 1 -- 172.21.15.121:0/1477572896 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f000070d0 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.829+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c149710 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.830+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c06ab60 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.831+0000 7f6ef3fff700 1 -- 172.21.15.121:0/1477572896 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6f00007480 con 0x7f6f0c0634b0 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6ef3fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6ef3fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:08:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6ef3fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:59.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6ef3fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:08:59.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6ef3fff700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6ef8098780 0x7f6ef809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:08:59.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6f0bfff700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6ef8098780 0x7f6ef809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:08:59.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6ef3fff700 1 -- 172.21.15.121:0/1477572896 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f6f000bfd60 con 0x7f6f0c0634b0 2023-09-29T23:08:59.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.832+0000 7f6f0bfff700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6ef8098780 0x7f6ef809ac30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009260 tx=0x7f6efc016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:08:59.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:08:59.835+0000 7f6ef3fff700 1 -- 172.21.15.121:0/1477572896 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f0008d380 con 0x7f6f0c0634b0 2023-09-29T23:09:00.055 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.056+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6f0c09dd30 con 0x7f6f0c0634b0 2023-09-29T23:09:00.056 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.056+0000 7f6ef3fff700 1 -- 172.21.15.121:0/1477572896 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6f00017020 con 0x7f6f0c0634b0 2023-09-29T23:09:00.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:00.056 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:00.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 10 mgrc shutdown 2023-09-29T23:09:00.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6ef8098780 msgr2=0x7f6ef809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:00.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6ef8098780 0x7f6ef809ac30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009260 tx=0x7f6efc016000 comp rx=0 tx=0).stop 2023-09-29T23:09:00.058 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0634b0 msgr2=0x7f6f0c14bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0634b0 0x7f6f0c14bde0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c7c0 tx=0x7f6f00007f70 comp rx=0 tx=0).stop 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 shutdown_connections 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6ef8098780 0x7f6ef809ac30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f0c09f580 0x7f6f0c097420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f0c0630e0 0x7f6f0c095d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 --2- 172.21.15.121:0/1477572896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f0c0634b0 0x7f6f0c14bde0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 >> 172.21.15.121:0/1477572896 conn(0x7f6f0c069e90 msgr2=0x7f6f0c0b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 shutdown_connections 2023-09-29T23:09:00.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:00.058+0000 7f6f1298a700 1 -- 172.21.15.121:0/1477572896 wait complete. 2023-09-29T23:09:03.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.200+0000 7f7b29df7700 1 Processor -- start 2023-09-29T23:09:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.200+0000 7f7b29df7700 1 -- start start 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b29df7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7b240683b0 0x7f7b24141150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b29df7700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b240a9ac0 con 0x7f7b24067fe0 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b29df7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b24140d00 con 0x7f7b240a25f0 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b29df7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b24140a30 con 0x7f7b240683b0 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b237fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7b24067fe0 0x7f7b240a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:36518/0 (socket says 172.21.15.121:36518) 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b237fe700 1 -- 172.21.15.121:0/3328699402 learned_addr learned my addr 172.21.15.121:0/3328699402 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b22ffd700 1 --2- 172.21.15.121:0/3328699402 >> v2:172.21.15.123:3300/0 conn(0x7f7b240683b0 0x7f7b24141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066370355 0 0) 0x7f7b240a9ac0 con 0x7f7b24067fe0 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.201+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c003570 con 0x7f7b24067fe0 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4115659921 0 0) 0x7f7b0c003570 con 0x7f7b24067fe0 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 >> v2:172.21.15.123:3300/0 conn(0x7f7b240683b0 msgr2=0x7f7b24141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 --2- 172.21.15.121:0/3328699402 >> v2:172.21.15.123:3300/0 conn(0x7f7b240683b0 0x7f7b24141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 >> v1:172.21.15.123:6789/0 conn(0x7f7b240a25f0 legacy=0x7f7b24141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240a2f30 con 0x7f7b24067fe0 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7f7b18001470 con 0x7f7b24067fe0 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 >> v1:172.21.15.121:6789/0 conn(0x7f7b24067fe0 legacy=0x7f7b240a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b0c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b0c004810 0x7f7b0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b0c0051c0 0x7f7b0c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b24140a30 con 0x7f7b0c0051c0 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c00c050 con 0x7f7b0c004810 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c00c210 con 0x7f7b0c003a60 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b237fe700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b0c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b23fff700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b0c0051c0 0x7f7b0c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b22ffd700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b0c004810 0x7f7b0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b237fe700 1 -- 172.21.15.121:0/3328699402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b0c004810 msgr2=0x7f7b0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b237fe700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b0c004810 0x7f7b0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b237fe700 1 -- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b0c0051c0 msgr2=0x7f7b0c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b237fe700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b0c0051c0 0x7f7b0c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b237fe700 1 -- 172.21.15.121:0/3328699402 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c003570 con 0x7f7b0c003a60 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b23fff700 1 -- 172.21.15.121:0/3328699402 reap_dead start 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b237fe700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b0c003e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c300 tx=0x7f7b18001df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a47aae2d529a15e9 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1800cb80 con 0x7f7b0c003a60 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1800cd00 con 0x7f7b0c003a60 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.202+0000 7f7b227fc700 1 -- 172.21.15.121:0/3328699402 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18005b70 con 0x7f7b0c003a60 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.203+0000 7f7b29df7700 1 -- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 msgr2=0x7f7b0c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.203+0000 7f7b29df7700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b0c003e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c300 tx=0x7f7b18001df0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.203+0000 7f7b29df7700 1 -- 172.21.15.121:0/3328699402 shutdown_connections 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.203+0000 7f7b29df7700 1 --2- 172.21.15.121:0/3328699402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b0c003e30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.203+0000 7f7b29df7700 1 -- 172.21.15.121:0/3328699402 >> 172.21.15.121:0/3328699402 conn(0x7f7b240b7720 msgr2=0x7f7b240678d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.203+0000 7f7b29df7700 1 -- 172.21.15.121:0/3328699402 shutdown_connections 2023-09-29T23:09:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.203+0000 7f7b29df7700 1 -- 172.21.15.121:0/3328699402 wait complete. 2023-09-29T23:09:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.204+0000 7f7b29df7700 1 Processor -- start 2023-09-29T23:09:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.204+0000 7f7b29df7700 1 -- start start 2023-09-29T23:09:03.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b240aa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b240aba10 0x7f7b24143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b241435a0 0x7f7b2413c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240ab230 con 0x7f7b241435a0 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b24143a30 con 0x7f7b240aba10 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b24143cf0 con 0x7f7b0c003a60 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b237fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b240aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b23fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b241435a0 0x7f7b2413c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b237fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b240aa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:54114/0 (socket says 172.21.15.121:54114) 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b237fe700 1 -- 172.21.15.121:0/1872194430 learned_addr learned my addr 172.21.15.121:0/1872194430 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b22ffd700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b240aba10 0x7f7b24143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b23fff700 1 -- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 msgr2=0x7f7b240aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b23fff700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b240aa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b23fff700 1 -- 172.21.15.121:0/1872194430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b240aba10 msgr2=0x7f7b24143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b23fff700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b240aba10 0x7f7b24143060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b23fff700 1 -- 172.21.15.121:0/1872194430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240abe40 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b23fff700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b241435a0 0x7f7b2413c8c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b240a1dc0 tx=0x7f7b1c0017d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b0b7fe700 1 -- 172.21.15.121:0/1872194430 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c006db0 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b0b7fe700 1 -- 172.21.15.121:0/1872194430 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c00c540 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b240ac360 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b0b7fe700 1 -- 172.21.15.121:0/1872194430 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00c940 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.205+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2413ce00 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.206+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b24069060 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.208+0000 7f7b0b7fe700 1 -- 172.21.15.121:0/1872194430 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7b1c007030 con 0x7f7b241435a0 2023-09-29T23:09:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.208+0000 7f7b0b7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.208+0000 7f7b0b7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.208+0000 7f7b0b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.208+0000 7f7b0b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.208+0000 7f7b0b7fe700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7b10098780 0x7f7b1009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.209+0000 7f7b237fe700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7b10098780 0x7f7b1009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.209+0000 7f7b0b7fe700 1 -- 172.21.15.121:0/1872194430 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f7b1c0bfb60 con 0x7f7b241435a0 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.209+0000 7f7b237fe700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7b10098780 0x7f7b1009ac30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b18001c80 tx=0x7f7b18015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.212+0000 7f7b0b7fe700 1 -- 172.21.15.121:0/1872194430 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b1c08d170 con 0x7f7b241435a0 2023-09-29T23:09:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.433+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b24071660 con 0x7f7b241435a0 2023-09-29T23:09:03.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.433+0000 7f7b0b7fe700 1 -- 172.21.15.121:0/1872194430 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7b1c090790 con 0x7f7b241435a0 2023-09-29T23:09:03.433 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:03.433 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 10 mgrc shutdown 2023-09-29T23:09:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7b10098780 msgr2=0x7f7b1009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7b10098780 0x7f7b1009ac30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b18001c80 tx=0x7f7b18015000 comp rx=0 tx=0).stop 2023-09-29T23:09:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b241435a0 msgr2=0x7f7b2413c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b241435a0 0x7f7b2413c8c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b240a1dc0 tx=0x7f7b1c0017d0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 shutdown_connections 2023-09-29T23:09:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f7b10098780 0x7f7b1009ac30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b241435a0 0x7f7b2413c8c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b0c003a60 0x7f7b240aa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 --2- 172.21.15.121:0/1872194430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b240aba10 0x7f7b24143060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 >> 172.21.15.121:0/1872194430 conn(0x7f7b240b7720 msgr2=0x7f7b240a5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 shutdown_connections 2023-09-29T23:09:03.436 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:03.435+0000 7f7b29df7700 1 -- 172.21.15.121:0/1872194430 wait complete. 2023-09-29T23:09:06.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:06.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.576+0000 7f21096c3700 1 Processor -- start 2023-09-29T23:09:06.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.576+0000 7f21096c3700 1 -- start start 2023-09-29T23:09:06.578 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21096c3700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21040b7560 0x7f21040a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21096c3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a1fe0 0x7f210413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21096c3700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f21040a2450 0x7f21041400e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21096c3700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21040a0fb0 con 0x7f21040b7560 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21096c3700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21040a3650 con 0x7f21040a2450 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21096c3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21040a97a0 con 0x7f21040a1fe0 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21027fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a1fe0 0x7f210413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f2102ffd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21040b7560 0x7f21040a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f2102ffd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21040b7560 0x7f21040a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40146/0 (socket says 172.21.15.121:40146) 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21027fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a1fe0 0x7f210413fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:33690/0 (socket says 172.21.15.121:33690) 2023-09-29T23:09:06.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f2102ffd700 1 -- 172.21.15.121:0/185042175 learned_addr learned my addr 172.21.15.121:0/185042175 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.577+0000 7f21037fe700 1 --2- 172.21.15.121:0/185042175 >> v2:172.21.15.123:3300/0 conn(0x7f21040a2450 0x7f21041400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2102ffd700 1 -- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a1fe0 msgr2=0x7f210413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2102ffd700 1 --2- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a1fe0 0x7f210413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2102ffd700 1 -- 172.21.15.121:0/185042175 >> v2:172.21.15.123:3300/0 conn(0x7f21040a2450 msgr2=0x7f21041400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2102ffd700 1 --2- 172.21.15.121:0/185042175 >> v2:172.21.15.123:3300/0 conn(0x7f21040a2450 0x7f21041400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2102ffd700 1 -- 172.21.15.121:0/185042175 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21040a9a60 con 0x7f21040b7560 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21027fc700 1 --2- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a1fe0 0x7f210413fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21037fe700 1 --2- 172.21.15.121:0/185042175 >> v2:172.21.15.123:3300/0 conn(0x7f21040a2450 0x7f21041400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2102ffd700 1 --2- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a0a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2104141960 tx=0x7f20f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f8d378f6280fd server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2101ffb700 1 -- 172.21.15.121:0/185042175 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8009e70 con 0x7f21040b7560 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2101ffb700 1 -- 172.21.15.121:0/185042175 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f800b860 con 0x7f21040b7560 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f2101ffb700 1 -- 172.21.15.121:0/185042175 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f800bc00 con 0x7f21040b7560 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 -- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 msgr2=0x7f21040a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 --2- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a0a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2104141960 tx=0x7f20f8009250 comp rx=0 tx=0).stop 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 -- 172.21.15.121:0/185042175 shutdown_connections 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 --2- 172.21.15.121:0/185042175 >> v2:172.21.15.123:3300/0 conn(0x7f21040a2450 0x7f21041400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 --2- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a0a70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 --2- 172.21.15.121:0/185042175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a1fe0 0x7f210413fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 -- 172.21.15.121:0/185042175 >> 172.21.15.121:0/185042175 conn(0x7f2104068350 msgr2=0x7f21040b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 -- 172.21.15.121:0/185042175 shutdown_connections 2023-09-29T23:09:06.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.578+0000 7f21096c3700 1 -- 172.21.15.121:0/185042175 wait complete. 2023-09-29T23:09:06.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 Processor -- start 2023-09-29T23:09:06.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 -- start start 2023-09-29T23:09:06.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21040a1fe0 0x7f21040aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a2450 0x7f21041454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21040a6940 con 0x7f21040b7560 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2104145ee0 con 0x7f21040a1fe0 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21096c3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210413d150 con 0x7f21040a2450 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21027fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a2450 0x7f21041454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21027fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a2450 0x7f21041454a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:33692/0 (socket says 172.21.15.121:33692) 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f21027fc700 1 -- 172.21.15.121:0/1255384652 learned_addr learned my addr 172.21.15.121:0/1255384652 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:06.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.580+0000 7f2102ffd700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21040a1fe0 0x7f21040aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21037fe700 1 -- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a2450 msgr2=0x7f21041454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21037fe700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a2450 0x7f21041454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21037fe700 1 -- 172.21.15.121:0/1255384652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21040a1fe0 msgr2=0x7f21040aa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21037fe700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21040a1fe0 0x7f21040aa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21037fe700 1 -- 172.21.15.121:0/1255384652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4009420 con 0x7f21040b7560 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21027fc700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a2450 0x7f21041454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21037fe700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a4f90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00cf00 tx=0x7f20fc007840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f20eaffd700 1 -- 172.21.15.121:0/1255384652 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc015440 con 0x7f21040b7560 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f20eaffd700 1 -- 172.21.15.121:0/1255384652 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc0155a0 con 0x7f21040b7560 2023-09-29T23:09:06.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f80094f0 con 0x7f21040b7560 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f20eaffd700 1 -- 172.21.15.121:0/1255384652 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc015960 con 0x7f21040b7560 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.581+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21040a6640 con 0x7f21040b7560 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.582+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2104071660 con 0x7f21040b7560 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f20eaffd700 1 -- 172.21.15.121:0/1255384652 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f20fc00d440 con 0x7f21040b7560 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f20eaffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f20eaffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f20eaffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f20eaffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f20eaffd700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f20f0098780 0x7f20f009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f2102ffd700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f20f0098780 0x7f20f009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.583+0000 7f20eaffd700 1 -- 172.21.15.121:0/1255384652 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f20fc0c0f50 con 0x7f21040b7560 2023-09-29T23:09:06.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.584+0000 7f2102ffd700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f20f0098780 0x7f20f009ac30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2104141960 tx=0x7f20f8017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:06.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.588+0000 7f20eaffd700 1 -- 172.21.15.121:0/1255384652 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f20fc08e600 con 0x7f21040b7560 2023-09-29T23:09:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.806+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f21040b7930 con 0x7f21040b7560 2023-09-29T23:09:06.806 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.806+0000 7f20eaffd700 1 -- 172.21.15.121:0/1255384652 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f20fc091c20 con 0x7f21040b7560 2023-09-29T23:09:06.806 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:06.806 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 10 mgrc shutdown 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f20f0098780 msgr2=0x7f20f009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f20f0098780 0x7f20f009ac30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2104141960 tx=0x7f20f8017000 comp rx=0 tx=0).stop 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 msgr2=0x7f21040a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a4f90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00cf00 tx=0x7f20fc007840 comp rx=0 tx=0).stop 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 shutdown_connections 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f20f0098780 0x7f20f009ac30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.808 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040b7560 0x7f21040a4f90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21040a1fe0 0x7f21040aa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 --2- 172.21.15.121:0/1255384652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040a2450 0x7f21041454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 >> 172.21.15.121:0/1255384652 conn(0x7f2104068350 msgr2=0x7f21040a0830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 shutdown_connections 2023-09-29T23:09:06.809 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:06.808+0000 7f21096c3700 1 -- 172.21.15.121:0/1255384652 wait complete. 2023-09-29T23:09:09.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:09.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.949+0000 7f21e6bf6700 1 Processor -- start 2023-09-29T23:09:09.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.949+0000 7f21e6bf6700 1 -- start start 2023-09-29T23:09:09.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e6bf6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e00630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:09.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e6bf6700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f21e00b3cf0 0x7f21e014b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:09.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e6bf6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e0141810 con 0x7f21e0141440 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e6bf6700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e009a7f0 con 0x7f21e00b3cf0 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e6bf6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00a4df0 con 0x7f21e009b770 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e4992700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f21e0141440 0x7f21e00a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:48020/0 (socket says 172.21.15.121:48020) 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21dffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e00630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e4992700 1 -- 172.21.15.121:0/3563563380 learned_addr learned my addr 172.21.15.121:0/3563563380 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21e5193700 1 --2- 172.21.15.121:0/3563563380 >> v2:172.21.15.123:3300/0 conn(0x7f21e00b3cf0 0x7f21e014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21dffff700 1 -- 172.21.15.121:0/3563563380 >> v2:172.21.15.123:3300/0 conn(0x7f21e00b3cf0 msgr2=0x7f21e014b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21dffff700 1 --2- 172.21.15.121:0/3563563380 >> v2:172.21.15.123:3300/0 conn(0x7f21e00b3cf0 0x7f21e014b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21df7fe700 1 -- 172.21.15.121:0/3563563380 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864754999 0 0) 0x7f21e0141810 con 0x7f21e0141440 2023-09-29T23:09:09.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21dffff700 1 -- 172.21.15.121:0/3563563380 >> v1:172.21.15.121:6789/0 conn(0x7f21e0141440 legacy=0x7f21e00a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:09.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21dffff700 1 -- 172.21.15.121:0/3563563380 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e00a50b0 con 0x7f21e009b770 2023-09-29T23:09:09.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21dffff700 1 --2- 172.21.15.121:0/3563563380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e00630e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0098e0 tx=0x7f21cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb5ac5211b1fed8e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:09.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21df7fe700 1 -- 172.21.15.121:0/3563563380 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc009e80 con 0x7f21e009b770 2023-09-29T23:09:09.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21df7fe700 1 -- 172.21.15.121:0/3563563380 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc00b860 con 0x7f21e009b770 2023-09-29T23:09:09.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.950+0000 7f21df7fe700 1 -- 172.21.15.121:0/3563563380 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc00bbe0 con 0x7f21e009b770 2023-09-29T23:09:09.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 -- 172.21.15.121:0/3563563380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 msgr2=0x7f21e00630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:09.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/3563563380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e00630e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0098e0 tx=0x7f21cc0091a0 comp rx=0 tx=0).stop 2023-09-29T23:09:09.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 -- 172.21.15.121:0/3563563380 shutdown_connections 2023-09-29T23:09:09.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/3563563380 >> v2:172.21.15.123:3300/0 conn(0x7f21e00b3cf0 0x7f21e014b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:09.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/3563563380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e00630e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:09.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 -- 172.21.15.121:0/3563563380 >> 172.21.15.121:0/3563563380 conn(0x7f21e0069e90 msgr2=0x7f21e00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:09.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 -- 172.21.15.121:0/3563563380 shutdown_connections 2023-09-29T23:09:09.956 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.951+0000 7f21e6bf6700 1 -- 172.21.15.121:0/3563563380 wait complete. 2023-09-29T23:09:09.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.952+0000 7f21e6bf6700 1 Processor -- start 2023-09-29T23:09:09.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.952+0000 7f21e6bf6700 1 -- start start 2023-09-29T23:09:09.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e0143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:09.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21e00b3cf0 0x7f21e0141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:09.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e0141440 0x7f21e0150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:09.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00a2f00 con 0x7f21e00b3cf0 2023-09-29T23:09:09.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e0142e40 con 0x7f21e0141440 2023-09-29T23:09:09.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e0150620 con 0x7f21e009b770 2023-09-29T23:09:09.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21e00b3cf0 0x7f21e0141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:09.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21e00b3cf0 0x7f21e0141bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40158/0 (socket says 172.21.15.121:40158) 2023-09-29T23:09:09.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 -- 172.21.15.121:0/1678250758 learned_addr learned my addr 172.21.15.121:0/1678250758 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:09.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e4992700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e0143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:09.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e5193700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e0141440 0x7f21e0150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:09.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 -- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 msgr2=0x7f21e0143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:09.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e0143ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:09.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 -- 172.21.15.121:0/1678250758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e0141440 msgr2=0x7f21e0150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:09.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e0141440 0x7f21e0150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:09.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 -- 172.21.15.121:0/1678250758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc009420 con 0x7f21e00b3cf0 2023-09-29T23:09:09.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e4992700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e0143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dffff700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21e00b3cf0 0x7f21e0141bf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f21cc013000 tx=0x7f21cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dcff9700 1 -- 172.21.15.121:0/1678250758 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc01f070 con 0x7f21e00b3cf0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dcff9700 1 -- 172.21.15.121:0/1678250758 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc00b9d0 con 0x7f21e00b3cf0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e0150340 con 0x7f21e00b3cf0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21dcff9700 1 -- 172.21.15.121:0/1678250758 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc011670 con 0x7f21e00b3cf0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.953+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e01449b0 con 0x7f21e00b3cf0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.954+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e0144ba0 con 0x7f21e00b3cf0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.955+0000 7f21dcff9700 1 -- 172.21.15.121:0/1678250758 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f21cc011aa0 con 0x7f21e00b3cf0 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.955+0000 7f21dcff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.956+0000 7f21dcff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:09.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.956+0000 7f21dcff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:09.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.956+0000 7f21dcff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:09.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.956+0000 7f21dcff9700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21c4098780 0x7f21c409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:09.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.956+0000 7f21e4992700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21c4098780 0x7f21c409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:09.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.956+0000 7f21dcff9700 1 -- 172.21.15.121:0/1678250758 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f21cc0c1890 con 0x7f21e00b3cf0 2023-09-29T23:09:09.961 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.956+0000 7f21e4992700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21c4098780 0x7f21c409ac30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21d4001d90 tx=0x7f21d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:09.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:09.960+0000 7f21dcff9700 1 -- 172.21.15.121:0/1678250758 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21cc08eea0 con 0x7f21e00b3cf0 2023-09-29T23:09:10.180 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.181+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f21e009c140 con 0x7f21e00b3cf0 2023-09-29T23:09:10.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.181+0000 7f21dcff9700 1 -- 172.21.15.121:0/1678250758 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f21cc028080 con 0x7f21e00b3cf0 2023-09-29T23:09:10.181 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:10.181 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:10.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 10 mgrc shutdown 2023-09-29T23:09:10.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21c4098780 msgr2=0x7f21c409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:10.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21c4098780 0x7f21c409ac30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21d4001d90 tx=0x7f21d4009040 comp rx=0 tx=0).stop 2023-09-29T23:09:10.183 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21e00b3cf0 msgr2=0x7f21e0141bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21e00b3cf0 0x7f21e0141bf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f21cc013000 tx=0x7f21cc017000 comp rx=0 tx=0).stop 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 shutdown_connections 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f21c4098780 0x7f21c409ac30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e0141440 0x7f21e0150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21e009b770 0x7f21e0143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 --2- 172.21.15.121:0/1678250758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21e00b3cf0 0x7f21e0141bf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 >> 172.21.15.121:0/1678250758 conn(0x7f21e0069e90 msgr2=0x7f21e00a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 shutdown_connections 2023-09-29T23:09:10.185 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:10.183+0000 7f21e6bf6700 1 -- 172.21.15.121:0/1678250758 wait complete. 2023-09-29T23:09:13.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:13.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.322+0000 7fe630f0e700 1 Processor -- start 2023-09-29T23:09:13.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.322+0000 7fe630f0e700 1 -- start start 2023-09-29T23:09:13.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe630f0e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe62c141440 0x7fe62c0a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:13.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe630f0e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c14b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe630f0e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62c141810 con 0x7fe62c09b770 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe630f0e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c09a7f0 con 0x7fe62c141440 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe630f0e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0a4df0 con 0x7fe62c0b3cf0 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c14b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe629d9b700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe62c09b770 0x7fe62c0630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:48036/0 (socket says 172.21.15.121:48036) 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.323+0000 7fe629d9b700 1 -- 172.21.15.121:0/269152784 learned_addr learned my addr 172.21.15.121:0/269152784 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62a59c700 1 --2- 172.21.15.121:0/269152784 >> v2:172.21.15.123:3300/0 conn(0x7fe62c141440 0x7fe62c0a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62ad9d700 1 -- 172.21.15.121:0/269152784 >> v2:172.21.15.123:3300/0 conn(0x7fe62c141440 msgr2=0x7fe62c0a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62959a700 1 -- 172.21.15.121:0/269152784 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873723901 0 0) 0x7fe62c141810 con 0x7fe62c09b770 2023-09-29T23:09:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62ad9d700 1 --2- 172.21.15.121:0/269152784 >> v2:172.21.15.123:3300/0 conn(0x7fe62c141440 0x7fe62c0a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62ad9d700 1 -- 172.21.15.121:0/269152784 >> v1:172.21.15.121:6789/0 conn(0x7fe62c09b770 legacy=0x7fe62c0630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62ad9d700 1 -- 172.21.15.121:0/269152784 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c0a50b0 con 0x7fe62c0b3cf0 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62ad9d700 1 --2- 172.21.15.121:0/269152784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c14b190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe624009e60 tx=0x7fe624009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec16f4d6fdf08f74 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62959a700 1 -- 172.21.15.121:0/269152784 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62400e070 con 0x7fe62c0b3cf0 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62959a700 1 -- 172.21.15.121:0/269152784 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62400bc30 con 0x7fe62c0b3cf0 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe62959a700 1 -- 172.21.15.121:0/269152784 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624012430 con 0x7fe62c0b3cf0 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe630f0e700 1 -- 172.21.15.121:0/269152784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 msgr2=0x7fe62c14b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.324+0000 7fe630f0e700 1 --2- 172.21.15.121:0/269152784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c14b190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe624009e60 tx=0x7fe624009780 comp rx=0 tx=0).stop 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.325+0000 7fe630f0e700 1 -- 172.21.15.121:0/269152784 shutdown_connections 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.325+0000 7fe630f0e700 1 --2- 172.21.15.121:0/269152784 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c14b190 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.325+0000 7fe630f0e700 1 --2- 172.21.15.121:0/269152784 >> v2:172.21.15.123:3300/0 conn(0x7fe62c141440 0x7fe62c0a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.325+0000 7fe630f0e700 1 -- 172.21.15.121:0/269152784 >> 172.21.15.121:0/269152784 conn(0x7fe62c069e90 msgr2=0x7fe62c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.325+0000 7fe630f0e700 1 -- 172.21.15.121:0/269152784 shutdown_connections 2023-09-29T23:09:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.325+0000 7fe630f0e700 1 -- 172.21.15.121:0/269152784 wait complete. 2023-09-29T23:09:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.325+0000 7fe630f0e700 1 Processor -- start 2023-09-29T23:09:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe630f0e700 1 -- start start 2023-09-29T23:09:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe630f0e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62c09b770 0x7fe62c150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe630f0e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c141e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe630f0e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe62c141440 0x7fe62c1423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:13.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe630f0e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0a2c10 con 0x7fe62c141440 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe630f0e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c142930 con 0x7fe62c09b770 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe630f0e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c150ce0 con 0x7fe62c0b3cf0 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe629d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe62c141440 0x7fe62c1423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe629d9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c141e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:33732/0 (socket says 172.21.15.121:33732) 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe629d9b700 1 -- 172.21.15.121:0/3356086461 learned_addr learned my addr 172.21.15.121:0/3356086461 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe62a59c700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62c09b770 0x7fe62c150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe62ad9d700 1 -- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 msgr2=0x7fe62c141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe62ad9d700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c141e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe62ad9d700 1 -- 172.21.15.121:0/3356086461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62c09b770 msgr2=0x7fe62c150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:13.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe62ad9d700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62c09b770 0x7fe62c150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe62ad9d700 1 -- 172.21.15.121:0/3356086461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624009a20 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.326+0000 7fe629d9b700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.327+0000 7fe62ad9d700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe62c141440 0x7fe62c1423c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe624016000 tx=0x7fe624019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.327+0000 7fe61a7fc700 1 -- 172.21.15.121:0/3356086461 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624007000 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.327+0000 7fe61a7fc700 1 -- 172.21.15.121:0/3356086461 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624025440 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.327+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62c142d70 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.327+0000 7fe61a7fc700 1 -- 172.21.15.121:0/3356086461 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624025800 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.327+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62c1510e0 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.328+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62c06ab60 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.329+0000 7fe61a7fc700 1 -- 172.21.15.121:0/3356086461 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe624003050 con 0x7fe62c141440 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.329+0000 7fe61a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:13.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.329+0000 7fe61a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.329+0000 7fe61a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.329+0000 7fe61a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.329+0000 7fe61a7fc700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe60c0987b0 0x7fe60c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.329+0000 7fe62a59c700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe60c0987b0 0x7fe60c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.330+0000 7fe61a7fc700 1 -- 172.21.15.121:0/3356086461 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fe6240c2c00 con 0x7fe62c141440 2023-09-29T23:09:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.330+0000 7fe62a59c700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe60c0987b0 0x7fe60c09ac60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe6200061f0 tx=0x7fe620008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:13.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.333+0000 7fe61a7fc700 1 -- 172.21.15.121:0/3356086461 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6240902b0 con 0x7fe62c141440 2023-09-29T23:09:13.556 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.555+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe62c071660 con 0x7fe62c141440 2023-09-29T23:09:13.556 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.555+0000 7fe61a7fc700 1 -- 172.21.15.121:0/3356086461 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe6240938d0 con 0x7fe62c141440 2023-09-29T23:09:13.556 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:13.556 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:13.558 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 10 mgrc shutdown 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe60c0987b0 msgr2=0x7fe60c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe60c0987b0 0x7fe60c09ac60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe6200061f0 tx=0x7fe620008040 comp rx=0 tx=0).stop 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe62c141440 msgr2=0x7fe62c1423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe62c141440 0x7fe62c1423c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe624016000 tx=0x7fe624019000 comp rx=0 tx=0).stop 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 shutdown_connections 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe60c0987b0 0x7fe60c09ac60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe62c141440 0x7fe62c1423c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.559 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62c09b770 0x7fe62c150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 --2- 172.21.15.121:0/3356086461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe62c0b3cf0 0x7fe62c141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:13.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 >> 172.21.15.121:0/3356086461 conn(0x7fe62c069e90 msgr2=0x7fe62c13ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:13.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 shutdown_connections 2023-09-29T23:09:13.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:13.557+0000 7fe630f0e700 1 -- 172.21.15.121:0/3356086461 wait complete. 2023-09-29T23:09:16.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:16.700 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.699+0000 7f61bec95700 1 Processor -- start 2023-09-29T23:09:16.700 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.699+0000 7f61bec95700 1 -- start start 2023-09-29T23:09:16.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bec95700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b806a3e0 0x7f61b80a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bec95700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f61b806a7b0 0x7f61b806e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bec95700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b8140700 con 0x7f61b806f5b0 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bec95700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b80a68c0 con 0x7f61b806a7b0 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bec95700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80a66f0 con 0x7f61b806a3e0 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bca31700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b806a3e0 0x7f61b80a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61b7fff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f61b806f5b0 0x7f61b8140df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38508/0 (socket says 172.21.15.121:38508) 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61b7fff700 1 -- 172.21.15.121:0/1259064025 learned_addr learned my addr 172.21.15.121:0/1259064025 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:16.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bd232700 1 --2- 172.21.15.121:0/1259064025 >> v2:172.21.15.123:3300/0 conn(0x7f61b806a7b0 0x7f61b806e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bca31700 1 -- 172.21.15.121:0/1259064025 >> v2:172.21.15.123:3300/0 conn(0x7f61b806a7b0 msgr2=0x7f61b806e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61b77fe700 1 -- 172.21.15.121:0/1259064025 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258993873 0 0) 0x7f61b8140700 con 0x7f61b806f5b0 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bca31700 1 --2- 172.21.15.121:0/1259064025 >> v2:172.21.15.123:3300/0 conn(0x7f61b806a7b0 0x7f61b806e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bca31700 1 -- 172.21.15.121:0/1259064025 >> v1:172.21.15.121:6789/0 conn(0x7f61b806f5b0 legacy=0x7f61b8140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bca31700 1 -- 172.21.15.121:0/1259064025 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b81409a0 con 0x7f61b806a3e0 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.700+0000 7f61bd232700 1 --2- 172.21.15.121:0/1259064025 >> v2:172.21.15.123:3300/0 conn(0x7f61b806a7b0 0x7f61b806e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bca31700 1 --2- 172.21.15.121:0/1259064025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b806a3e0 0x7f61b80a61b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0098e0 tx=0x7f61ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3dae2e0ff62844f server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61b77fe700 1 -- 172.21.15.121:0/1259064025 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac009e80 con 0x7f61b806a3e0 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61b77fe700 1 -- 172.21.15.121:0/1259064025 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac00b860 con 0x7f61b806a3e0 2023-09-29T23:09:16.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61b77fe700 1 -- 172.21.15.121:0/1259064025 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac00bbe0 con 0x7f61b806a3e0 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 -- 172.21.15.121:0/1259064025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b806a3e0 msgr2=0x7f61b80a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 --2- 172.21.15.121:0/1259064025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b806a3e0 0x7f61b80a61b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f61ac0098e0 tx=0x7f61ac0091a0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 -- 172.21.15.121:0/1259064025 shutdown_connections 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 --2- 172.21.15.121:0/1259064025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b806a3e0 0x7f61b80a61b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 --2- 172.21.15.121:0/1259064025 >> v2:172.21.15.123:3300/0 conn(0x7f61b806a7b0 0x7f61b806e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 -- 172.21.15.121:0/1259064025 >> 172.21.15.121:0/1259064025 conn(0x7f61b8069020 msgr2=0x7f61b8069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 -- 172.21.15.121:0/1259064025 shutdown_connections 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.701+0000 7f61bec95700 1 -- 172.21.15.121:0/1259064025 wait complete. 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.702+0000 7f61bec95700 1 Processor -- start 2023-09-29T23:09:16.708 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bec95700 1 -- start start 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bec95700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61b806a3e0 0x7f61b814e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bec95700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b806a7b0 0x7f61b814ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bec95700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b8098830 0x7f61b8098cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bec95700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b814f350 con 0x7f61b806a3e0 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bec95700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8098050 con 0x7f61b806a7b0 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bec95700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80a13c0 con 0x7f61b8098830 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bca31700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61b806a3e0 0x7f61b814e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bca31700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61b806a3e0 0x7f61b814e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37988/0 (socket says 172.21.15.121:37988) 2023-09-29T23:09:16.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bca31700 1 -- 172.21.15.121:0/3165074093 learned_addr learned my addr 172.21.15.121:0/3165074093 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bd232700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b8098830 0x7f61b8098cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61b7fff700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b806a7b0 0x7f61b814ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bca31700 1 -- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b8098830 msgr2=0x7f61b8098cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bca31700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b8098830 0x7f61b8098cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bca31700 1 -- 172.21.15.121:0/3165074093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b806a7b0 msgr2=0x7f61b814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.703+0000 7f61bca31700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b806a7b0 0x7f61b814ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61bca31700 1 -- 172.21.15.121:0/3165074093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac009420 con 0x7f61b806a3e0 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61b7fff700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b806a7b0 0x7f61b814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61bca31700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61b806a3e0 0x7f61b814e8d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00b3c0 tx=0x7f61ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:16.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61b4ff9700 1 -- 172.21.15.121:0/3165074093 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac01f070 con 0x7f61b806a3e0 2023-09-29T23:09:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61b4ff9700 1 -- 172.21.15.121:0/3165074093 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac00b9d0 con 0x7f61b806a3e0 2023-09-29T23:09:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b814f540 con 0x7f61b806a3e0 2023-09-29T23:09:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61b4ff9700 1 -- 172.21.15.121:0/3165074093 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac011650 con 0x7f61b806a3e0 2023-09-29T23:09:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.704+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b80a3000 con 0x7f61b806a3e0 2023-09-29T23:09:16.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.705+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b8071660 con 0x7f61b806a3e0 2023-09-29T23:09:16.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.706+0000 7f61b4ff9700 1 -- 172.21.15.121:0/3165074093 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f61ac011a00 con 0x7f61b806a3e0 2023-09-29T23:09:16.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.706+0000 7f61b4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.706+0000 7f61b4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.706+0000 7f61b4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.706+0000 7f61b4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.706+0000 7f61b4ff9700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f619c0987b0 0x7f619c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.707+0000 7f61b7fff700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f619c0987b0 0x7f619c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.707+0000 7f61b4ff9700 1 -- 172.21.15.121:0/3165074093 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f61ac0c1ca0 con 0x7f61b806a3e0 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.707+0000 7f61b7fff700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f619c0987b0 0x7f619c09ac60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f61b806a300 tx=0x7f61a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.710+0000 7f61b4ff9700 1 -- 172.21.15.121:0/3165074093 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61ac08f350 con 0x7f61b806a3e0 2023-09-29T23:09:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.928+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61b806ed60 con 0x7f61b806a3e0 2023-09-29T23:09:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.928+0000 7f61b4ff9700 1 -- 172.21.15.121:0/3165074093 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f61ac011cb0 con 0x7f61b806a3e0 2023-09-29T23:09:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:16.929 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:16.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 10 mgrc shutdown 2023-09-29T23:09:16.931 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f619c0987b0 msgr2=0x7f619c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f619c0987b0 0x7f619c09ac60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f61b806a300 tx=0x7f61a4009040 comp rx=0 tx=0).stop 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61b806a3e0 msgr2=0x7f61b814e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61b806a3e0 0x7f61b814e8d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00b3c0 tx=0x7f61ac017000 comp rx=0 tx=0).stop 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 shutdown_connections 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f619c0987b0 0x7f619c09ac60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61b8098830 0x7f61b8098cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61b806a3e0 0x7f61b814e8d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 --2- 172.21.15.121:0/3165074093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b806a7b0 0x7f61b814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 >> 172.21.15.121:0/3165074093 conn(0x7f61b8069020 msgr2=0x7f61b80a46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:16.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 shutdown_connections 2023-09-29T23:09:16.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:16.930+0000 7f61bec95700 1 -- 172.21.15.121:0/3165074093 wait complete. 2023-09-29T23:09:19.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa841331700 1 Processor -- start 2023-09-29T23:09:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa841331700 1 -- start start 2023-09-29T23:09:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa841331700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c0b4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa841331700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa83c133d20 0x7fa83c1341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa841331700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83c0b39d0 con 0x7fa83c0a6940 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa841331700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83c0a6d10 con 0x7fa83c133d20 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa841331700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0b37f0 con 0x7fa83c0b3e60 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa83a7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c0b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa83affd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa83c0a6940 0x7fa83c0aa060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38520/0 (socket says 172.21.15.121:38520) 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa83b7fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa83c133d20 0x7fa83c1341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.071+0000 7fa83affd700 1 -- 172.21.15.121:0/530761594 learned_addr learned my addr 172.21.15.121:0/530761594 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:20.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa83a7fc700 1 -- 172.21.15.121:0/530761594 >> v2:172.21.15.121:3300/0 conn(0x7fa83c133d20 msgr2=0x7fa83c1341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa83a7fc700 1 --2- 172.21.15.121:0/530761594 >> v2:172.21.15.121:3300/0 conn(0x7fa83c133d20 0x7fa83c1341a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa83a7fc700 1 -- 172.21.15.121:0/530761594 >> v1:172.21.15.121:6789/0 conn(0x7fa83c0a6940 legacy=0x7fa83c0aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa83a7fc700 1 -- 172.21.15.121:0/530761594 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c0a9410 con 0x7fa83c0b3e60 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa83b7fe700 1 --2- 172.21.15.121:0/530761594 >> v2:172.21.15.121:3300/0 conn(0x7fa83c133d20 0x7fa83c1341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa83a7fc700 1 --2- 172.21.15.121:0/530761594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c0b4250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0098e0 tx=0x7fa82c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6651841f4a5915c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa839ffb700 1 -- 172.21.15.121:0/530761594 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00b640 con 0x7fa83c0b3e60 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa839ffb700 1 -- 172.21.15.121:0/530761594 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c00b7a0 con 0x7fa83c0b3e60 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa839ffb700 1 -- 172.21.15.121:0/530761594 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00bb40 con 0x7fa83c0b3e60 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa841331700 1 -- 172.21.15.121:0/530761594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 msgr2=0x7fa83c0b4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa841331700 1 --2- 172.21.15.121:0/530761594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c0b4250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0098e0 tx=0x7fa82c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa841331700 1 -- 172.21.15.121:0/530761594 shutdown_connections 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa841331700 1 --2- 172.21.15.121:0/530761594 >> v2:172.21.15.121:3300/0 conn(0x7fa83c133d20 0x7fa83c1341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.072+0000 7fa841331700 1 --2- 172.21.15.121:0/530761594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c0b4250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.073+0000 7fa841331700 1 -- 172.21.15.121:0/530761594 >> 172.21.15.121:0/530761594 conn(0x7fa83c06c010 msgr2=0x7fa83c06ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.073+0000 7fa841331700 1 -- 172.21.15.121:0/530761594 shutdown_connections 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.073+0000 7fa841331700 1 -- 172.21.15.121:0/530761594 wait complete. 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 Processor -- start 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 -- start start 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa83c0a6940 0x7fa83c0aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:20.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c06d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa83c133d20 0x7fa83c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0adb60 con 0x7fa83c0a6940 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0ab5d0 con 0x7fa83c133d20 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa841331700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0ab890 con 0x7fa83c0b3e60 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa83affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa83c0a6940 0x7fa83c0aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa83a7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c06d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa83affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa83c0a6940 0x7fa83c0aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38010/0 (socket says 172.21.15.121:38010) 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa83a7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c06d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58232/0 (socket says 172.21.15.121:58232) 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.074+0000 7fa83affd700 1 -- 172.21.15.121:0/1475463427 learned_addr learned my addr 172.21.15.121:0/1475463427 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:20.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83a7fc700 1 -- 172.21.15.121:0/1475463427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa83c133d20 msgr2=0x7fa83c06db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83b7fe700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa83c133d20 0x7fa83c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83a7fc700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa83c133d20 0x7fa83c06db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83a7fc700 1 -- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa83c0a6940 msgr2=0x7fa83c0aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83a7fc700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa83c0a6940 0x7fa83c0aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83a7fc700 1 -- 172.21.15.121:0/1475463427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834009a20 con 0x7fa83c0b3e60 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83affd700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa83c0a6940 0x7fa83c0aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa83a7fc700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c06d620 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa82c016000 tx=0x7fa82c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa822ffd700 1 -- 172.21.15.121:0/1475463427 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c01bc20 con 0x7fa83c0b3e60 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa822ffd700 1 -- 172.21.15.121:0/1475463427 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c01bd80 con 0x7fa83c0b3e60 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c009420 con 0x7fa83c0b3e60 2023-09-29T23:09:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa822ffd700 1 -- 172.21.15.121:0/1475463427 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c011670 con 0x7fa83c0b3e60 2023-09-29T23:09:20.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.075+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83c06ea30 con 0x7fa83c0b3e60 2023-09-29T23:09:20.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.076+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83c0b4330 con 0x7fa83c0b3e60 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.077+0000 7fa822ffd700 1 -- 172.21.15.121:0/1475463427 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fa82c011a20 con 0x7fa83c0b3e60 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.077+0000 7fa822ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.077+0000 7fa822ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.077+0000 7fa822ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.077+0000 7fa822ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.077+0000 7fa822ffd700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa8280987b0 0x7fa82809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.078+0000 7fa83affd700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa8280987b0 0x7fa82809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:20.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.078+0000 7fa822ffd700 1 -- 172.21.15.121:0/1475463427 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fa82c0c1ce0 con 0x7fa83c0b3e60 2023-09-29T23:09:20.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.078+0000 7fa83affd700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa8280987b0 0x7fa82809ac60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa830004900 tx=0x7fa830007f70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:20.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.081+0000 7fa822ffd700 1 -- 172.21.15.121:0/1475463427 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa82c08f390 con 0x7fa83c0b3e60 2023-09-29T23:09:20.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.300+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa83c06ebc0 con 0x7fa83c0b3e60 2023-09-29T23:09:20.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.300+0000 7fa822ffd700 1 -- 172.21.15.121:0/1475463427 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa82c0929b0 con 0x7fa83c0b3e60 2023-09-29T23:09:20.302 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:20.303 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:20.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 10 mgrc shutdown 2023-09-29T23:09:20.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa8280987b0 msgr2=0x7fa82809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa8280987b0 0x7fa82809ac60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa830004900 tx=0x7fa830007f70 comp rx=0 tx=0).stop 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 msgr2=0x7fa83c06d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c06d620 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa82c016000 tx=0x7fa82c017000 comp rx=0 tx=0).stop 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 shutdown_connections 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa8280987b0 0x7fa82809ac60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa83c133d20 0x7fa83c06db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa83c0a6940 0x7fa83c0aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 --2- 172.21.15.121:0/1475463427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa83c0b3e60 0x7fa83c06d620 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 >> 172.21.15.121:0/1475463427 conn(0x7fa83c06c010 msgr2=0x7fa83c142680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 shutdown_connections 2023-09-29T23:09:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:20.302+0000 7fa841331700 1 -- 172.21.15.121:0/1475463427 wait complete. 2023-09-29T23:09:23.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:23.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecf69d9700 1 Processor -- start 2023-09-29T23:09:23.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecf69d9700 1 -- start start 2023-09-29T23:09:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecf69d9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fecf006a630 0x7fecf006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecf69d9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fecf01438a0 0x7fecf0143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecf69d9700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf006aa00 con 0x7fecf0144530 2023-09-29T23:09:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecf69d9700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf01406b0 con 0x7fecf01438a0 2023-09-29T23:09:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecf69d9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf006df40 con 0x7fecf006a630 2023-09-29T23:09:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.444+0000 7fecef7fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fecf01438a0 0x7fecf0143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7fecef7fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fecf01438a0 0x7fecf0143ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38026/0 (socket says 172.21.15.121:38026) 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7fecef7fe700 1 -- 172.21.15.121:0/3605018757 learned_addr learned my addr 172.21.15.121:0/3605018757 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7feceffff700 1 --2- 172.21.15.121:0/3605018757 >> v2:172.21.15.123:3300/0 conn(0x7fecf006a630 0x7fecf006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7fecef7fe700 1 -- 172.21.15.121:0/3605018757 >> v2:172.21.15.123:3300/0 conn(0x7fecf006a630 msgr2=0x7fecf006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7fecef7fe700 1 --2- 172.21.15.121:0/3605018757 >> v2:172.21.15.123:3300/0 conn(0x7fecf006a630 0x7fecf006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7fecef7fe700 1 -- 172.21.15.121:0/3605018757 >> v1:172.21.15.123:6789/0 conn(0x7fecf0144530 legacy=0x7fecf0140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7fecef7fe700 1 -- 172.21.15.121:0/3605018757 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf01402b0 con 0x7fecf01438a0 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7feceffff700 1 --2- 172.21.15.121:0/3605018757 >> v2:172.21.15.123:3300/0 conn(0x7fecf006a630 0x7fecf006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7fecef7fe700 1 --2- 172.21.15.121:0/3605018757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf01438a0 0x7fecf0143ff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fecf00684f0 tx=0x7fece4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c4afd39ed7c654 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7feceeffd700 1 -- 172.21.15.121:0/3605018757 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400b860 con 0x7fecf01438a0 2023-09-29T23:09:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7feceeffd700 1 -- 172.21.15.121:0/3605018757 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece400b9c0 con 0x7fecf01438a0 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.445+0000 7feceeffd700 1 -- 172.21.15.121:0/3605018757 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400bd60 con 0x7fecf01438a0 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 -- 172.21.15.121:0/3605018757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf01438a0 msgr2=0x7fecf0143ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 --2- 172.21.15.121:0/3605018757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf01438a0 0x7fecf0143ff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fecf00684f0 tx=0x7fece4009250 comp rx=0 tx=0).stop 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 -- 172.21.15.121:0/3605018757 shutdown_connections 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 --2- 172.21.15.121:0/3605018757 >> v2:172.21.15.123:3300/0 conn(0x7fecf006a630 0x7fecf006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 --2- 172.21.15.121:0/3605018757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf01438a0 0x7fecf0143ff0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 -- 172.21.15.121:0/3605018757 >> 172.21.15.121:0/3605018757 conn(0x7fecf0068fd0 msgr2=0x7fecf0069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 -- 172.21.15.121:0/3605018757 shutdown_connections 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.446+0000 7fecf69d9700 1 -- 172.21.15.121:0/3605018757 wait complete. 2023-09-29T23:09:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 Processor -- start 2023-09-29T23:09:23.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 -- start start 2023-09-29T23:09:23.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fecf006a630 0x7fecf009f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:23.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecf01438a0 0x7fecf0150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:23.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0144530 0x7fecf01507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:23.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf009fb10 con 0x7fecf0144530 2023-09-29T23:09:23.452 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf009eb80 con 0x7fecf01438a0 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.447+0000 7fecf69d9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf0150da0 con 0x7fecf006a630 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7feceffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fecf006a630 0x7fecf009f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf4f76700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0144530 0x7fecf01507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7feceffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fecf006a630 0x7fecf009f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58240/0 (socket says 172.21.15.121:58240) 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7feceffff700 1 -- 172.21.15.121:0/146122023 learned_addr learned my addr 172.21.15.121:0/146122023 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecef7fe700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecf01438a0 0x7fecf0150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf4f76700 1 -- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fecf006a630 msgr2=0x7fecf009f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf4f76700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fecf006a630 0x7fecf009f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf4f76700 1 -- 172.21.15.121:0/146122023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecf01438a0 msgr2=0x7fecf0150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf4f76700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecf01438a0 0x7fecf0150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf4f76700 1 -- 172.21.15.121:0/146122023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40094f0 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7feceffff700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fecf006a630 0x7fecf009f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf4f76700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0144530 0x7fecf01507a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fecf006e560 tx=0x7fece80015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecd7fff700 1 -- 172.21.15.121:0/146122023 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8011590 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecd7fff700 1 -- 172.21.15.121:0/146122023 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece8011710 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf01510c0 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecd7fff700 1 -- 172.21.15.121:0/146122023 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8011ad0 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.448+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf009bac0 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.449+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf0071660 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.450+0000 7fecd7fff700 1 -- 172.21.15.121:0/146122023 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fece8003030 con 0x7fecf0144530 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.450+0000 7fecd7fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.451+0000 7fecd7fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.451+0000 7fecd7fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.451+0000 7fecd7fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.451+0000 7fecd7fff700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecdc098780 0x7fecdc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.451+0000 7feceffff700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecdc098780 0x7fecdc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.451+0000 7fecd7fff700 1 -- 172.21.15.121:0/146122023 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fece80bfe40 con 0x7fecf0144530 2023-09-29T23:09:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.451+0000 7feceffff700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecdc098780 0x7fecdc09ac30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fece0006dc0 tx=0x7fece0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.455+0000 7fecd7fff700 1 -- 172.21.15.121:0/146122023 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fece808d4f0 con 0x7fecf0144530 2023-09-29T23:09:23.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.675+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fecf006e2d0 con 0x7fecf0144530 2023-09-29T23:09:23.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.676+0000 7fecd7fff700 1 -- 172.21.15.121:0/146122023 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fece8090b10 con 0x7fecf0144530 2023-09-29T23:09:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:23.677 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 10 mgrc shutdown 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecdc098780 msgr2=0x7fecdc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecdc098780 0x7fecdc09ac30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fece0006dc0 tx=0x7fece0008040 comp rx=0 tx=0).stop 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0144530 msgr2=0x7fecf01507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0144530 0x7fecf01507a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fecf006e560 tx=0x7fece80015f0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 shutdown_connections 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecdc098780 0x7fecdc09ac30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecf0144530 0x7fecf01507a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fecf006a630 0x7fecf009f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 --2- 172.21.15.121:0/146122023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecf01438a0 0x7fecf0150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 >> 172.21.15.121:0/146122023 conn(0x7fecf0068fd0 msgr2=0x7fecf00a5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 shutdown_connections 2023-09-29T23:09:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:23.678+0000 7fecf69d9700 1 -- 172.21.15.121:0/146122023 wait complete. 2023-09-29T23:09:26.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:26.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.819+0000 7f57412d2700 1 Processor -- start 2023-09-29T23:09:26.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.819+0000 7f57412d2700 1 -- start start 2023-09-29T23:09:26.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f57412d2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c067690 0x7f573c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:26.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f57412d2700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f573c06d750 0x7f573c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:26.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f57412d2700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c067a60 con 0x7f573c069d70 2023-09-29T23:09:26.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f57412d2700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c06e760 con 0x7f573c06d750 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f57412d2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c06aa00 con 0x7f573c067690 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573affd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c067690 0x7f573c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f573c06d750 0x7f573c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f573c06d750 0x7f573c06db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58070/0 (socket says 172.21.15.121:58070) 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573affd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c067690 0x7f573c0a5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58124/0 (socket says 172.21.15.121:58124) 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 -- 172.21.15.121:0/2297406612 learned_addr learned my addr 172.21.15.121:0/2297406612 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 -- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c067690 msgr2=0x7f573c0a5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 --2- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c067690 0x7f573c0a5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 -- 172.21.15.121:0/2297406612 >> v1:172.21.15.123:6789/0 conn(0x7f573c069d70 legacy=0x7f573c06a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 -- 172.21.15.121:0/2297406612 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c06e0a0 con 0x7f573c06d750 2023-09-29T23:09:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573affd700 1 --2- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c067690 0x7f573c0a5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f573a7fc700 1 --2- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c06d750 0x7f573c06db60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f572c0099b0 tx=0x7f572c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae31ea2fe54701bf server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f5739ffb700 1 -- 172.21.15.121:0/2297406612 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c009ea0 con 0x7f573c06d750 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f5739ffb700 1 -- 172.21.15.121:0/2297406612 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c00b860 con 0x7f573c06d750 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.820+0000 7f5739ffb700 1 -- 172.21.15.121:0/2297406612 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00bc00 con 0x7f573c06d750 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 -- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c06d750 msgr2=0x7f573c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 --2- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c06d750 0x7f573c06db60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f572c0099b0 tx=0x7f572c009250 comp rx=0 tx=0).stop 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 -- 172.21.15.121:0/2297406612 shutdown_connections 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 --2- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c067690 0x7f573c0a5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 --2- 172.21.15.121:0/2297406612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c06d750 0x7f573c06db60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 -- 172.21.15.121:0/2297406612 >> 172.21.15.121:0/2297406612 conn(0x7f573c068550 msgr2=0x7f573c068950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 -- 172.21.15.121:0/2297406612 shutdown_connections 2023-09-29T23:09:26.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.821+0000 7f57412d2700 1 -- 172.21.15.121:0/2297406612 wait complete. 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.822+0000 7f57412d2700 1 Processor -- start 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.822+0000 7f57412d2700 1 -- start start 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f57412d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c067690 0x7f573c0a3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f57412d2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c069d70 0x7f573c1534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f57412d2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f573c06d750 0x7f573c09c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f57412d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c153cc0 con 0x7f573c067690 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f57412d2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c153f80 con 0x7f573c06d750 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f57412d2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c153a00 con 0x7f573c069d70 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c067690 0x7f573c0a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:26.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c069d70 0x7f573c1534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c069d70 0x7f573c1534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58134/0 (socket says 172.21.15.121:58134) 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 -- 172.21.15.121:0/4164357047 learned_addr learned my addr 172.21.15.121:0/4164357047 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573b7fe700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f573c06d750 0x7f573c09c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 -- 172.21.15.121:0/4164357047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f573c06d750 msgr2=0x7f573c09c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f573c06d750 0x7f573c09c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 -- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c067690 msgr2=0x7f573c0a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c067690 0x7f573c0a3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 -- 172.21.15.121:0/4164357047 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730009420 con 0x7f573c069d70 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573b7fe700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f573c06d750 0x7f573c09c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573affd700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c067690 0x7f573c0a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:26.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f573a7fc700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c069d70 0x7f573c1534c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f573c0693d0 tx=0x7f572c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f5722ffd700 1 -- 172.21.15.121:0/4164357047 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00bbe0 con 0x7f573c069d70 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f5722ffd700 1 -- 172.21.15.121:0/4164357047 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c011420 con 0x7f573c069d70 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c0094f0 con 0x7f573c069d70 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.823+0000 7f5722ffd700 1 -- 172.21.15.121:0/4164357047 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c0117c0 con 0x7f573c069d70 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.824+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c09a2b0 con 0x7f573c069d70 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.824+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c071660 con 0x7f573c069d70 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.826+0000 7f5722ffd700 1 -- 172.21.15.121:0/4164357047 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f572c011bf0 con 0x7f573c069d70 2023-09-29T23:09:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.826+0000 7f5722ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:26.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.826+0000 7f5722ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:26.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.826+0000 7f5722ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:26.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.826+0000 7f5722ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:26.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.826+0000 7f5722ffd700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5728098780 0x7f572809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:26.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.827+0000 7f573affd700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5728098780 0x7f572809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:26.830 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.827+0000 7f5722ffd700 1 -- 172.21.15.121:0/4164357047 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f572c0c24b0 con 0x7f573c069d70 2023-09-29T23:09:26.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.827+0000 7f573affd700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5728098780 0x7f572809ac30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5730011600 tx=0x7f57300091a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:26.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:26.830+0000 7f5722ffd700 1 -- 172.21.15.121:0/4164357047 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f572c08fb60 con 0x7f573c069d70 2023-09-29T23:09:27.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.050+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f573c0a4300 con 0x7f573c069d70 2023-09-29T23:09:27.051 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.050+0000 7f5722ffd700 1 -- 172.21.15.121:0/4164357047 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f572c093180 con 0x7f573c069d70 2023-09-29T23:09:27.051 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:27.052 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.052+0000 7f57412d2700 10 mgrc shutdown 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.052+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5728098780 msgr2=0x7f572809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.052+0000 7f57412d2700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5728098780 0x7f572809ac30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5730011600 tx=0x7f57300091a0 comp rx=0 tx=0).stop 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.052+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c069d70 msgr2=0x7f573c1534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.052+0000 7f57412d2700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c069d70 0x7f573c1534c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f573c0693d0 tx=0x7f572c017000 comp rx=0 tx=0).stop 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.052+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 shutdown_connections 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.052+0000 7f57412d2700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5728098780 0x7f572809ac30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:27.054 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.053+0000 7f57412d2700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f573c06d750 0x7f573c09c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:27.055 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.053+0000 7f57412d2700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c067690 0x7f573c0a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:27.055 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.053+0000 7f57412d2700 1 --2- 172.21.15.121:0/4164357047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f573c069d70 0x7f573c1534c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:27.055 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.053+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 >> 172.21.15.121:0/4164357047 conn(0x7f573c068550 msgr2=0x7f573c068920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:27.055 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.053+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 shutdown_connections 2023-09-29T23:09:27.055 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:27.053+0000 7f57412d2700 1 -- 172.21.15.121:0/4164357047 wait complete. 2023-09-29T23:09:30.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:30.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.194+0000 7f73905aa700 1 Processor -- start 2023-09-29T23:09:30.195 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.194+0000 7f73905aa700 1 -- start start 2023-09-29T23:09:30.197 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f73905aa700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7388141440 0x7f73880a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f73905aa700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f73880630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f73905aa700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73880b3cf0 0x7f7388143df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f73905aa700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7388141810 con 0x7f73880b3cf0 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f73905aa700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738809a7f0 con 0x7f7388141440 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f73905aa700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73880a4df0 con 0x7f738809b770 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738db45700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f73880630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738eb47700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73880b3cf0 0x7f7388143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738db45700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f73880630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58140/0 (socket says 172.21.15.121:58140) 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738eb47700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73880b3cf0 0x7f7388143df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58086/0 (socket says 172.21.15.121:58086) 2023-09-29T23:09:30.198 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738db45700 1 -- 172.21.15.121:0/307386069 learned_addr learned my addr 172.21.15.121:0/307386069 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738e346700 1 --2- 172.21.15.121:0/307386069 >> v2:172.21.15.123:3300/0 conn(0x7f7388141440 0x7f73880a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738eb47700 1 -- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 msgr2=0x7f73880630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738eb47700 1 --2- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f73880630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738eb47700 1 -- 172.21.15.121:0/307386069 >> v2:172.21.15.123:3300/0 conn(0x7f7388141440 msgr2=0x7f73880a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738eb47700 1 --2- 172.21.15.121:0/307386069 >> v2:172.21.15.123:3300/0 conn(0x7f7388141440 0x7f73880a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738eb47700 1 -- 172.21.15.121:0/307386069 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880a50b0 con 0x7f73880b3cf0 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.195+0000 7f738db45700 1 --2- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f73880630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f738eb47700 1 --2- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73880b3cf0 0x7f7388143df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f738806bea0 tx=0x7f7380009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3992c774407f0388 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f738d344700 1 -- 172.21.15.121:0/307386069 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738000e070 con 0x7f73880b3cf0 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f738d344700 1 -- 172.21.15.121:0/307386069 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738000bb60 con 0x7f73880b3cf0 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f738d344700 1 -- 172.21.15.121:0/307386069 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380012430 con 0x7f73880b3cf0 2023-09-29T23:09:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 -- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73880b3cf0 msgr2=0x7f7388143df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 --2- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73880b3cf0 0x7f7388143df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f738806bea0 tx=0x7f7380009780 comp rx=0 tx=0).stop 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 -- 172.21.15.121:0/307386069 shutdown_connections 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 --2- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73880b3cf0 0x7f7388143df0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 --2- 172.21.15.121:0/307386069 >> v2:172.21.15.123:3300/0 conn(0x7f7388141440 0x7f73880a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 --2- 172.21.15.121:0/307386069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f73880630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 -- 172.21.15.121:0/307386069 >> 172.21.15.121:0/307386069 conn(0x7f7388069e90 msgr2=0x7f73880b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 -- 172.21.15.121:0/307386069 shutdown_connections 2023-09-29T23:09:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.196+0000 7f73905aa700 1 -- 172.21.15.121:0/307386069 wait complete. 2023-09-29T23:09:30.203 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.197+0000 7f73905aa700 1 Processor -- start 2023-09-29T23:09:30.203 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.197+0000 7f73905aa700 1 -- start start 2023-09-29T23:09:30.203 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f73905aa700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f738813b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:30.203 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f73905aa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73880b3cf0 0x7f7388141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f73905aa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7388141440 0x7f7388153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f73905aa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7388151860 con 0x7f7388141440 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f73905aa700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738813b750 con 0x7f73880b3cf0 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f73905aa700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7388154510 con 0x7f738809b770 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738eb47700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7388141440 0x7f7388153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738e346700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f738813b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738e346700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f738813b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58154/0 (socket says 172.21.15.121:58154) 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738e346700 1 -- 172.21.15.121:0/2419215936 learned_addr learned my addr 172.21.15.121:0/2419215936 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738db45700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73880b3cf0 0x7f7388141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:30.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738eb47700 1 -- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 msgr2=0x7f738813b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738eb47700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f738813b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738eb47700 1 -- 172.21.15.121:0/2419215936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73880b3cf0 msgr2=0x7f7388141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738eb47700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73880b3cf0 0x7f7388141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738eb47700 1 -- 172.21.15.121:0/2419215936 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73840094f0 con 0x7f7388141440 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738e346700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f738813b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.198+0000 7f738eb47700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7388141440 0x7f7388153c30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f738806b1b0 tx=0x7f7380019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.199+0000 7f737e7fc700 1 -- 172.21.15.121:0/2419215936 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73800032b0 con 0x7f7388141440 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.199+0000 7f737e7fc700 1 -- 172.21.15.121:0/2419215936 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73800070c0 con 0x7f7388141440 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.199+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7380009a20 con 0x7f7388141440 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.199+0000 7f737e7fc700 1 -- 172.21.15.121:0/2419215936 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380026620 con 0x7f7388141440 2023-09-29T23:09:30.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.199+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7388154a50 con 0x7f7388141440 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.200+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738806ab60 con 0x7f7388141440 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f737e7fc700 1 -- 172.21.15.121:0/2419215936 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7380026a50 con 0x7f7388141440 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f737e7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f737e7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f737e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f737e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f737e7fc700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f736c098780 0x7f736c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f738e346700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f736c098780 0x7f736c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f737e7fc700 1 -- 172.21.15.121:0/2419215936 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f73800c2520 con 0x7f7388141440 2023-09-29T23:09:30.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.201+0000 7f738e346700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f736c098780 0x7f736c09ac30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7378006aa0 tx=0x7f7378008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:30.207 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.204+0000 7f737e7fc700 1 -- 172.21.15.121:0/2419215936 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f738008fbd0 con 0x7f7388141440 2023-09-29T23:09:30.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.429+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7388071660 con 0x7f7388141440 2023-09-29T23:09:30.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.430+0000 7f737e7fc700 1 -- 172.21.15.121:0/2419215936 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f73800931f0 con 0x7f7388141440 2023-09-29T23:09:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:30.431 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:30.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 10 mgrc shutdown 2023-09-29T23:09:30.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f736c098780 msgr2=0x7f736c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:30.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f736c098780 0x7f736c09ac30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7378006aa0 tx=0x7f7378008040 comp rx=0 tx=0).stop 2023-09-29T23:09:30.433 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7388141440 msgr2=0x7f7388153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7388141440 0x7f7388153c30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f738806b1b0 tx=0x7f7380019000 comp rx=0 tx=0).stop 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 shutdown_connections 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f736c098780 0x7f736c09ac30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7388141440 0x7f7388153c30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f738809b770 0x7f738813b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 --2- 172.21.15.121:0/2419215936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73880b3cf0 0x7f7388141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 >> 172.21.15.121:0/2419215936 conn(0x7f7388069e90 msgr2=0x7f73880a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 shutdown_connections 2023-09-29T23:09:30.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:30.432+0000 7f73905aa700 1 -- 172.21.15.121:0/2419215936 wait complete. 2023-09-29T23:09:33.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:33.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.571+0000 7ff007e2f700 1 Processor -- start 2023-09-29T23:09:33.572 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.571+0000 7ff007e2f700 1 -- start start 2023-09-29T23:09:33.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.571+0000 7ff007e2f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0000630e0 0x7ff00009c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:33.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff007e2f700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff00009c700 0x7ff0000a1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:33.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff007e2f700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0000a0620 con 0x7ff0000b3cf0 2023-09-29T23:09:33.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff007e2f700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0000a08e0 con 0x7ff00009c700 2023-09-29T23:09:33.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff007e2f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0000a62d0 con 0x7ff0000630e0 2023-09-29T23:09:33.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0000630e0 0x7ff00009c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:33.574 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0000630e0 0x7ff00009c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58160/0 (socket says 172.21.15.121:58160) 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 -- 172.21.15.121:0/2277319002 learned_addr learned my addr 172.21.15.121:0/2277319002 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff0053ca700 1 --2- 172.21.15.121:0/2277319002 >> v2:172.21.15.123:3300/0 conn(0x7ff00009c700 0x7ff0000a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 -- 172.21.15.121:0/2277319002 >> v2:172.21.15.123:3300/0 conn(0x7ff00009c700 msgr2=0x7ff0000a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 --2- 172.21.15.121:0/2277319002 >> v2:172.21.15.123:3300/0 conn(0x7ff00009c700 0x7ff0000a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 -- 172.21.15.121:0/2277319002 >> v1:172.21.15.123:6789/0 conn(0x7ff0000b3cf0 legacy=0x7ff0000a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 -- 172.21.15.121:0/2277319002 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0000a02c0 con 0x7ff0000630e0 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff005bcb700 1 --2- 172.21.15.121:0/2277319002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0000630e0 0x7ff00009c1c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feffc009990 tx=0x7feffc009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb968f3044d5f64 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff004bc9700 1 -- 172.21.15.121:0/2277319002 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc005c10 con 0x7ff0000630e0 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff004bc9700 1 -- 172.21.15.121:0/2277319002 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc005d70 con 0x7ff0000630e0 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.572+0000 7ff004bc9700 1 -- 172.21.15.121:0/2277319002 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc00b840 con 0x7ff0000630e0 2023-09-29T23:09:33.575 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 -- 172.21.15.121:0/2277319002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0000630e0 msgr2=0x7ff00009c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:33.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 --2- 172.21.15.121:0/2277319002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0000630e0 0x7ff00009c1c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7feffc009990 tx=0x7feffc009220 comp rx=0 tx=0).stop 2023-09-29T23:09:33.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 -- 172.21.15.121:0/2277319002 shutdown_connections 2023-09-29T23:09:33.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 --2- 172.21.15.121:0/2277319002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0000630e0 0x7ff00009c1c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 --2- 172.21.15.121:0/2277319002 >> v2:172.21.15.123:3300/0 conn(0x7ff00009c700 0x7ff0000a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 -- 172.21.15.121:0/2277319002 >> 172.21.15.121:0/2277319002 conn(0x7ff000069e90 msgr2=0x7ff0000b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:33.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 -- 172.21.15.121:0/2277319002 shutdown_connections 2023-09-29T23:09:33.576 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.573+0000 7ff007e2f700 1 -- 172.21.15.121:0/2277319002 wait complete. 2023-09-29T23:09:33.578 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.574+0000 7ff007e2f700 1 Processor -- start 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.574+0000 7ff007e2f700 1 -- start start 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff007e2f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0000630e0 0x7ff00006d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff007e2f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff00009c700 0x7ff00014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff007e2f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0000b3cf0 0x7ff0000975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff007e2f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00006e560 con 0x7ff0000630e0 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff007e2f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00014e860 con 0x7ff0000b3cf0 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff007e2f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00014eb20 con 0x7ff00009c700 2023-09-29T23:09:33.579 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff00009c700 0x7ff00014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff005bcb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0000630e0 0x7ff00006d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff00009c700 0x7ff00014de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:58162/0 (socket says 172.21.15.121:58162) 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 -- 172.21.15.121:0/1960656195 learned_addr learned my addr 172.21.15.121:0/1960656195 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0063cc700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0000b3cf0 0x7ff0000975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 -- 172.21.15.121:0/1960656195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0000b3cf0 msgr2=0x7ff0000975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0000b3cf0 0x7ff0000975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 -- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0000630e0 msgr2=0x7ff00006d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0000630e0 0x7ff00006d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 -- 172.21.15.121:0/1960656195 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc009480 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff0053ca700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff00009c700 0x7ff00014de70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feff000d890 tx=0x7feff0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7feff5ffb700 1 -- 172.21.15.121:0/1960656195 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00125f0 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7feff5ffb700 1 -- 172.21.15.121:0/1960656195 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0012770 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00014ba80 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.575+0000 7feff5ffb700 1 -- 172.21.15.121:0/1960656195 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0012b30 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.576+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00006dce0 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.576+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff000096460 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.577+0000 7feff5ffb700 1 -- 172.21.15.121:0/1960656195 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7feff001e460 con 0x7ff00009c700 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.577+0000 7feff5ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:33.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.578+0000 7feff5ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:33.582 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.578+0000 7feff5ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:33.582 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.578+0000 7feff5ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:33.582 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.578+0000 7feff5ffb700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fefe4098780 0x7fefe409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:33.582 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.578+0000 7feff5ffb700 1 -- 172.21.15.121:0/1960656195 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7feff00c0ae0 con 0x7ff00009c700 2023-09-29T23:09:33.582 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.578+0000 7ff005bcb700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fefe4098780 0x7fefe409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:33.582 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.578+0000 7ff005bcb700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fefe4098780 0x7fefe409ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feffc0030c0 tx=0x7feffc015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:33.583 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.582+0000 7feff5ffb700 1 -- 172.21.15.121:0/1960656195 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff000096460 con 0x7ff00009c700 2023-09-29T23:09:33.801 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.800+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff000096460 con 0x7ff00009c700 2023-09-29T23:09:33.801 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.801+0000 7feff5ffb700 1 -- 172.21.15.121:0/1960656195 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff000096460 con 0x7ff00009c700 2023-09-29T23:09:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:33.802 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 10 mgrc shutdown 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fefe4098780 msgr2=0x7fefe409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fefe4098780 0x7fefe409ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feffc0030c0 tx=0x7feffc015000 comp rx=0 tx=0).stop 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff00009c700 msgr2=0x7ff00014de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff00009c700 0x7ff00014de70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feff000d890 tx=0x7feff0008040 comp rx=0 tx=0).stop 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 shutdown_connections 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fefe4098780 0x7fefe409ac30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0000b3cf0 0x7ff0000975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0000630e0 0x7ff00006d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.804 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 --2- 172.21.15.121:0/1960656195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff00009c700 0x7ff00014de70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:33.805 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 >> 172.21.15.121:0/1960656195 conn(0x7ff000069e90 msgr2=0x7ff0001421a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:33.805 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 shutdown_connections 2023-09-29T23:09:33.805 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:33.803+0000 7ff007e2f700 1 -- 172.21.15.121:0/1960656195 wait complete. 2023-09-29T23:09:36.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:36.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.942+0000 7fc69cae4700 1 Processor -- start 2023-09-29T23:09:36.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.942+0000 7fc69cae4700 1 -- start start 2023-09-29T23:09:36.944 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc69cae4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980a2180 0x7fc69814a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc69cae4700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6980a0620 con 0x7fc6980b3cf0 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc69cae4700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6980a08e0 con 0x7fc6980630e0 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc69cae4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69809c8d0 con 0x7fc6980a2180 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc695d9b700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc6980b3cf0 0x7fc698146900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52112/0 (socket says 172.21.15.121:52112) 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc695d9b700 1 -- 172.21.15.121:0/1086418015 learned_addr learned my addr 172.21.15.121:0/1086418015 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc696d9d700 1 --2- 172.21.15.121:0/1086418015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980a2180 0x7fc69814a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc69559a700 1 -- 172.21.15.121:0/1086418015 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147090108 0 0) 0x7fc6980a0620 con 0x7fc6980b3cf0 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc69559a700 1 -- 172.21.15.121:0/1086418015 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc680003570 con 0x7fc6980b3cf0 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.943+0000 7fc696d9d700 1 -- 172.21.15.121:0/1086418015 >> v1:172.21.15.123:6789/0 conn(0x7fc6980630e0 legacy=0x7fc69809c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc696d9d700 1 -- 172.21.15.121:0/1086418015 >> v1:172.21.15.121:6789/0 conn(0x7fc6980b3cf0 legacy=0x7fc698146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:36.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc696d9d700 1 -- 172.21.15.121:0/1086418015 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980a62d0 con 0x7fc6980a2180 2023-09-29T23:09:36.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc696d9d700 1 --2- 172.21.15.121:0/1086418015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980a2180 0x7fc69814a030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc690009e90 tx=0x7fc690009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba38dae9ec77d1d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:36.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69559a700 1 -- 172.21.15.121:0/1086418015 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69000e070 con 0x7fc6980a2180 2023-09-29T23:09:36.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69559a700 1 -- 172.21.15.121:0/1086418015 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69000bc30 con 0x7fc6980a2180 2023-09-29T23:09:36.946 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69559a700 1 -- 172.21.15.121:0/1086418015 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690012430 con 0x7fc6980a2180 2023-09-29T23:09:36.948 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69cae4700 1 -- 172.21.15.121:0/1086418015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980a2180 msgr2=0x7fc69814a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69cae4700 1 --2- 172.21.15.121:0/1086418015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980a2180 0x7fc69814a030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc690009e90 tx=0x7fc690009780 comp rx=0 tx=0).stop 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69cae4700 1 -- 172.21.15.121:0/1086418015 shutdown_connections 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69cae4700 1 --2- 172.21.15.121:0/1086418015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980a2180 0x7fc69814a030 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.944+0000 7fc69cae4700 1 -- 172.21.15.121:0/1086418015 >> 172.21.15.121:0/1086418015 conn(0x7fc698069e90 msgr2=0x7fc6980b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.945+0000 7fc69cae4700 1 -- 172.21.15.121:0/1086418015 shutdown_connections 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.945+0000 7fc69cae4700 1 -- 172.21.15.121:0/1086418015 wait complete. 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.945+0000 7fc69cae4700 1 Processor -- start 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.945+0000 7fc69cae4700 1 -- start start 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69cae4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6980630e0 0x7fc6980a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69cae4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6980a2180 0x7fc69809acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69cae4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980b3cf0 0x7fc69814dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:36.949 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69cae4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc698098e30 con 0x7fc6980630e0 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69cae4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69814ea70 con 0x7fc6980a2180 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69cae4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69814ed30 con 0x7fc6980b3cf0 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6980630e0 0x7fc6980a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc696d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980b3cf0 0x7fc69814dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6980630e0 0x7fc6980a7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51808/0 (socket says 172.21.15.121:51808) 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 -- 172.21.15.121:0/2042052724 learned_addr learned my addr 172.21.15.121:0/2042052724 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc695d9b700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6980a2180 0x7fc69809acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 -- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980b3cf0 msgr2=0x7fc69814dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980b3cf0 0x7fc69814dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:36.950 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 -- 172.21.15.121:0/2042052724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6980a2180 msgr2=0x7fc69809acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6980a2180 0x7fc69809acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc69659c700 1 -- 172.21.15.121:0/2042052724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc690009a20 con 0x7fc6980630e0 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.946+0000 7fc696d9d700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980b3cf0 0x7fc69814dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.947+0000 7fc69659c700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6980630e0 0x7fc6980a7190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00c790 tx=0x7fc68c001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.947+0000 7fc6867fc700 1 -- 172.21.15.121:0/2042052724 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c006870 con 0x7fc6980630e0 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.947+0000 7fc6867fc700 1 -- 172.21.15.121:0/2042052724 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c0069d0 con 0x7fc6980630e0 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.947+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69809b5c0 con 0x7fc6980630e0 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.947+0000 7fc6867fc700 1 -- 172.21.15.121:0/2042052724 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c006d90 con 0x7fc6980630e0 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.947+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69814e320 con 0x7fc6980630e0 2023-09-29T23:09:36.951 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.948+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69806ab60 con 0x7fc6980630e0 2023-09-29T23:09:36.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc6867fc700 1 -- 172.21.15.121:0/2042052724 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc68c007240 con 0x7fc6980630e0 2023-09-29T23:09:36.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc6867fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:36.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc6867fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:36.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc6867fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:36.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc6867fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:36.952 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc6867fc700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc678098780 0x7fc67809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:36.953 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc695d9b700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc678098780 0x7fc67809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:36.953 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.949+0000 7fc6867fc700 1 -- 172.21.15.121:0/2042052724 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fc68c0c00c0 con 0x7fc6980630e0 2023-09-29T23:09:36.953 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.950+0000 7fc695d9b700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc678098780 0x7fc67809ac30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc6980affa0 tx=0x7fc688009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:36.953 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:36.953+0000 7fc6867fc700 1 -- 172.21.15.121:0/2042052724 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc68c08d6e0 con 0x7fc6980630e0 2023-09-29T23:09:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.177+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc698071660 con 0x7fc6980630e0 2023-09-29T23:09:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.178+0000 7fc6867fc700 1 -- 172.21.15.121:0/2042052724 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc68c017080 con 0x7fc6980630e0 2023-09-29T23:09:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:37.179 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 10 mgrc shutdown 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc678098780 msgr2=0x7fc67809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc678098780 0x7fc67809ac30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc6980affa0 tx=0x7fc688009040 comp rx=0 tx=0).stop 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6980630e0 msgr2=0x7fc6980a7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6980630e0 0x7fc6980a7190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00c790 tx=0x7fc68c001170 comp rx=0 tx=0).stop 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 shutdown_connections 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc678098780 0x7fc67809ac30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6980b3cf0 0x7fc69814dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6980630e0 0x7fc6980a7190 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 --2- 172.21.15.121:0/2042052724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6980a2180 0x7fc69809acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 >> 172.21.15.121:0/2042052724 conn(0x7fc698069e90 msgr2=0x7fc6980b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 shutdown_connections 2023-09-29T23:09:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:37.180+0000 7fc69cae4700 1 -- 172.21.15.121:0/2042052724 wait complete. 2023-09-29T23:09:40.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:40.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.320+0000 7fd032052700 1 Processor -- start 2023-09-29T23:09:40.321 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.320+0000 7fd032052700 1 -- start start 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd032052700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0634b0 0x7fd02c1458f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd032052700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c09e600 con 0x7fd02c09f580 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd032052700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0a2d20 con 0x7fd02c0630e0 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd032052700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c09d8d0 con 0x7fd02c0634b0 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02affd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0634b0 0x7fd02c1458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02bfff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd02c09f580 0x7fd02c147e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52114/0 (socket says 172.21.15.121:52114) 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02bfff700 1 -- 172.21.15.121:0/2960410752 learned_addr learned my addr 172.21.15.121:0/2960410752 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02affd700 1 -- 172.21.15.121:0/2960410752 >> v1:172.21.15.123:6789/0 conn(0x7fd02c0630e0 legacy=0x7fd02c09c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02a7fc700 1 -- 172.21.15.121:0/2960410752 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429785446 0 0) 0x7fd02c09e600 con 0x7fd02c09f580 2023-09-29T23:09:40.323 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02affd700 1 -- 172.21.15.121:0/2960410752 >> v1:172.21.15.121:6789/0 conn(0x7fd02c09f580 legacy=0x7fd02c147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02affd700 1 -- 172.21.15.121:0/2960410752 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c09db70 con 0x7fd02c0634b0 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02affd700 1 --2- 172.21.15.121:0/2960410752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0634b0 0x7fd02c1458f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009910 tx=0x7fd01c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a833c0131a634e9 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02a7fc700 1 -- 172.21.15.121:0/2960410752 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00b820 con 0x7fd02c0634b0 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02a7fc700 1 -- 172.21.15.121:0/2960410752 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c00b980 con 0x7fd02c0634b0 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.321+0000 7fd02a7fc700 1 -- 172.21.15.121:0/2960410752 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00bd20 con 0x7fd02c0634b0 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.322+0000 7fd032052700 1 -- 172.21.15.121:0/2960410752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0634b0 msgr2=0x7fd02c1458f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.322+0000 7fd032052700 1 --2- 172.21.15.121:0/2960410752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0634b0 0x7fd02c1458f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009910 tx=0x7fd01c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.322+0000 7fd032052700 1 -- 172.21.15.121:0/2960410752 shutdown_connections 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.322+0000 7fd032052700 1 --2- 172.21.15.121:0/2960410752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0634b0 0x7fd02c1458f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.322+0000 7fd032052700 1 -- 172.21.15.121:0/2960410752 >> 172.21.15.121:0/2960410752 conn(0x7fd02c069e90 msgr2=0x7fd02c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:40.324 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.322+0000 7fd032052700 1 -- 172.21.15.121:0/2960410752 shutdown_connections 2023-09-29T23:09:40.325 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.322+0000 7fd032052700 1 -- 172.21.15.121:0/2960410752 wait complete. 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.323+0000 7fd032052700 1 Processor -- start 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.323+0000 7fd032052700 1 -- start start 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0630e0 0x7fd02c142130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02c0634b0 0x7fd02c0a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd02c09f580 0x7fd02c0a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c1413d0 con 0x7fd02c09f580 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c141600 con 0x7fd02c0634b0 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c140bd0 con 0x7fd02c0630e0 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0630e0 0x7fd02c142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:40.327 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd02c09f580 0x7fd02c0a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0630e0 0x7fd02c142130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:57898/0 (socket says 172.21.15.121:57898) 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 -- 172.21.15.121:0/1681355285 learned_addr learned my addr 172.21.15.121:0/1681355285 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02affd700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02c0634b0 0x7fd02c0a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 -- 172.21.15.121:0/1681355285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02c0634b0 msgr2=0x7fd02c0a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02c0634b0 0x7fd02c0a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 -- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd02c09f580 msgr2=0x7fd02c0a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd02c09f580 0x7fd02c0a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 -- 172.21.15.121:0/1681355285 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c009420 con 0x7fd02c0630e0 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd02b7fe700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0630e0 0x7fd02c142130 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd02000c5e0 tx=0x7fd0200020e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:40.328 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd0137fe700 1 -- 172.21.15.121:0/1681355285 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd020011590 con 0x7fd02c0630e0 2023-09-29T23:09:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd0137fe700 1 -- 172.21.15.121:0/1681355285 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0200116f0 con 0x7fd02c0630e0 2023-09-29T23:09:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c153a90 con 0x7fd02c0630e0 2023-09-29T23:09:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd0137fe700 1 -- 172.21.15.121:0/1681355285 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd020011a90 con 0x7fd02c0630e0 2023-09-29T23:09:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.324+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c153610 con 0x7fd02c0630e0 2023-09-29T23:09:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.325+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c141f80 con 0x7fd02c0630e0 2023-09-29T23:09:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.326+0000 7fd0137fe700 1 -- 172.21.15.121:0/1681355285 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd02001c460 con 0x7fd02c0630e0 2023-09-29T23:09:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.326+0000 7fd0137fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.327+0000 7fd0137fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.327+0000 7fd0137fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.327+0000 7fd0137fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.327+0000 7fd0137fe700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd018098780 0x7fd01809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.327+0000 7fd02affd700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd018098780 0x7fd01809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.327+0000 7fd0137fe700 1 -- 172.21.15.121:0/1681355285 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fd0200c02f0 con 0x7fd02c0630e0 2023-09-29T23:09:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.327+0000 7fd02affd700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd018098780 0x7fd01809ac30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0030c0 tx=0x7fd01c016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:40.332 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.331+0000 7fd0137fe700 1 -- 172.21.15.121:0/1681355285 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd02008d870 con 0x7fd02c0630e0 2023-09-29T23:09:40.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.556+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd02c140d10 con 0x7fd02c0630e0 2023-09-29T23:09:40.557 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.557+0000 7fd0137fe700 1 -- 172.21.15.121:0/1681355285 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd020090e90 con 0x7fd02c0630e0 2023-09-29T23:09:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:40.558 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 10 mgrc shutdown 2023-09-29T23:09:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd018098780 msgr2=0x7fd01809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd018098780 0x7fd01809ac30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0030c0 tx=0x7fd01c016000 comp rx=0 tx=0).stop 2023-09-29T23:09:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0630e0 msgr2=0x7fd02c142130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0630e0 0x7fd02c142130 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd02000c5e0 tx=0x7fd0200020e0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 shutdown_connections 2023-09-29T23:09:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd018098780 0x7fd01809ac30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd02c09f580 0x7fd02c0a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd02c0630e0 0x7fd02c142130 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 --2- 172.21.15.121:0/1681355285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd02c0634b0 0x7fd02c0a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 >> 172.21.15.121:0/1681355285 conn(0x7fd02c069e90 msgr2=0x7fd02c0b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 shutdown_connections 2023-09-29T23:09:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:40.559+0000 7fd032052700 1 -- 172.21.15.121:0/1681355285 wait complete. 2023-09-29T23:09:43.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:43.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.700+0000 7f47173c6700 1 Processor -- start 2023-09-29T23:09:43.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.700+0000 7f47173c6700 1 -- start start 2023-09-29T23:09:43.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f47173c6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f47100b3cf0 0x7f471014c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:43.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f47173c6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4710141810 con 0x7f471009b770 2023-09-29T23:09:43.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f47173c6700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471009a7f0 con 0x7f4710141440 2023-09-29T23:09:43.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f47173c6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100a4df0 con 0x7f47100b3cf0 2023-09-29T23:09:43.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4715963700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f47100b3cf0 0x7f471014c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:43.703 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4715963700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f47100b3cf0 0x7f471014c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51830/0 (socket says 172.21.15.121:51830) 2023-09-29T23:09:43.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4714961700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f471009b770 0x7f47100630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:52118/0 (socket says 172.21.15.121:52118) 2023-09-29T23:09:43.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4715963700 1 -- 172.21.15.121:0/3309160933 learned_addr learned my addr 172.21.15.121:0/3309160933 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:43.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4715963700 1 -- 172.21.15.121:0/3309160933 >> v1:172.21.15.123:6789/0 conn(0x7f4710141440 legacy=0x7f47100a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:43.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4715963700 1 -- 172.21.15.121:0/3309160933 >> v1:172.21.15.121:6789/0 conn(0x7f471009b770 legacy=0x7f47100630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:43.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4715963700 1 -- 172.21.15.121:0/3309160933 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100a50b0 con 0x7f47100b3cf0 2023-09-29T23:09:43.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4707fff700 1 -- 172.21.15.121:0/3309160933 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474704546 0 0) 0x7f4710141810 con 0x7f471009b770 2023-09-29T23:09:43.704 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4715963700 1 --2- 172.21.15.121:0/3309160933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47100b3cf0 0x7f471014c1a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4708009f20 tx=0x7f4708009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc59989a4b26b38 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4707fff700 1 -- 172.21.15.121:0/3309160933 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470800e070 con 0x7f47100b3cf0 2023-09-29T23:09:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4707fff700 1 -- 172.21.15.121:0/3309160933 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470800ba80 con 0x7f47100b3cf0 2023-09-29T23:09:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.701+0000 7f4707fff700 1 -- 172.21.15.121:0/3309160933 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47080092e0 con 0x7f47100b3cf0 2023-09-29T23:09:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.702+0000 7f47173c6700 1 -- 172.21.15.121:0/3309160933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47100b3cf0 msgr2=0x7f471014c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.702+0000 7f47173c6700 1 --2- 172.21.15.121:0/3309160933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47100b3cf0 0x7f471014c1a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4708009f20 tx=0x7f4708009840 comp rx=0 tx=0).stop 2023-09-29T23:09:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.702+0000 7f47173c6700 1 -- 172.21.15.121:0/3309160933 shutdown_connections 2023-09-29T23:09:43.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.702+0000 7f47173c6700 1 --2- 172.21.15.121:0/3309160933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47100b3cf0 0x7f471014c1a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:43.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.702+0000 7f47173c6700 1 -- 172.21.15.121:0/3309160933 >> 172.21.15.121:0/3309160933 conn(0x7f4710069e90 msgr2=0x7f47100b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:43.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.702+0000 7f47173c6700 1 -- 172.21.15.121:0/3309160933 shutdown_connections 2023-09-29T23:09:43.706 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.702+0000 7f47173c6700 1 -- 172.21.15.121:0/3309160933 wait complete. 2023-09-29T23:09:43.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.703+0000 7f47173c6700 1 Processor -- start 2023-09-29T23:09:43.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.703+0000 7f47173c6700 1 -- start start 2023-09-29T23:09:43.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f471009b770 0x7f471006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:43.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47100b3cf0 0x7f471009eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:43.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4710141440 0x7f471013ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:43.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100a0f40 con 0x7f4710141440 2023-09-29T23:09:43.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100a2e30 con 0x7f471009b770 2023-09-29T23:09:43.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100a1640 con 0x7f47100b3cf0 2023-09-29T23:09:43.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4710141440 0x7f471013ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:43.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4714961700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47100b3cf0 0x7f471009eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:43.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4710141440 0x7f471013ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51842/0 (socket says 172.21.15.121:51842) 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 -- 172.21.15.121:0/1948665102 learned_addr learned my addr 172.21.15.121:0/1948665102 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715162700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f471009b770 0x7f471006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 -- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47100b3cf0 msgr2=0x7f471009eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47100b3cf0 0x7f471009eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 -- 172.21.15.121:0/1948665102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f471009b770 msgr2=0x7f471006ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f471009b770 0x7f471006ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 -- 172.21.15.121:0/1948665102 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4708009ae0 con 0x7f4710141440 2023-09-29T23:09:43.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715162700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f471009b770 0x7f471006ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:09:43.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f4715963700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4710141440 0x7f471013ac50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f471006ad60 tx=0x7f4708018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:43.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47057fa700 1 -- 172.21.15.121:0/1948665102 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470800be90 con 0x7f4710141440 2023-09-29T23:09:43.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47057fa700 1 -- 172.21.15.121:0/1948665102 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4708007d80 con 0x7f4710141440 2023-09-29T23:09:43.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47100a1140 con 0x7f4710141440 2023-09-29T23:09:43.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47057fa700 1 -- 172.21.15.121:0/1948665102 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47080266d0 con 0x7f4710141440 2023-09-29T23:09:43.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.704+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471009fee0 con 0x7f4710141440 2023-09-29T23:09:43.712 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.705+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471006ab60 con 0x7f4710141440 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.706+0000 7f47057fa700 1 -- 172.21.15.121:0/1948665102 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f470801d420 con 0x7f4710141440 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.706+0000 7f47057fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.707+0000 7f47057fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.707+0000 7f47057fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.707+0000 7f47057fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.707+0000 7f47057fa700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f46f4098800 0x7f46f409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.707+0000 7f4715162700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f46f4098800 0x7f46f409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:43.713 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.707+0000 7f47057fa700 1 -- 172.21.15.121:0/1948665102 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f47080c2720 con 0x7f4710141440 2023-09-29T23:09:43.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.707+0000 7f4715162700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f46f4098800 0x7f46f409acb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f470c001850 tx=0x7f470c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:43.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.710+0000 7f47057fa700 1 -- 172.21.15.121:0/1948665102 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f470805e5a0 con 0x7f4710141440 2023-09-29T23:09:43.929 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.928+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4710071660 con 0x7f4710141440 2023-09-29T23:09:43.929 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.929+0000 7f47057fa700 1 -- 172.21.15.121:0/1948665102 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f470808fde0 con 0x7f4710141440 2023-09-29T23:09:43.930 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:43.930 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:43.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.930+0000 7f47173c6700 10 mgrc shutdown 2023-09-29T23:09:43.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.930+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f46f4098800 msgr2=0x7f46f409acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:43.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.930+0000 7f47173c6700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f46f4098800 0x7f46f409acb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f470c001850 tx=0x7f470c009040 comp rx=0 tx=0).stop 2023-09-29T23:09:43.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.930+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4710141440 msgr2=0x7f471013ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:43.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.930+0000 7f47173c6700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4710141440 0x7f471013ac50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f471006ad60 tx=0x7f4708018000 comp rx=0 tx=0).stop 2023-09-29T23:09:43.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 shutdown_connections 2023-09-29T23:09:43.932 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f46f4098800 0x7f46f409acb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:43.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4710141440 0x7f471013ac50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:43.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f471009b770 0x7f471006ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:43.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 --2- 172.21.15.121:0/1948665102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47100b3cf0 0x7f471009eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:43.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 >> 172.21.15.121:0/1948665102 conn(0x7f4710069e90 msgr2=0x7f47100630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:43.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 shutdown_connections 2023-09-29T23:09:43.933 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:43.931+0000 7f47173c6700 1 -- 172.21.15.121:0/1948665102 wait complete. 2023-09-29T23:09:46.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:47.074 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.073+0000 7fcaee163700 1 Processor -- start 2023-09-29T23:09:47.074 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.073+0000 7fcaee163700 1 -- start start 2023-09-29T23:09:47.076 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcaee163700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcae809f6a0 0x7fcae80a1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:47.076 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcaee163700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcae80a33e0 0x7fcae81475a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:47.076 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcaee163700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae80699d0 con 0x7fcae8069600 2023-09-29T23:09:47.076 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcaee163700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae80a18b0 con 0x7fcae809f6a0 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcaee163700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae809fc50 con 0x7fcae80a33e0 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcae77fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcae8069600 0x7fcae80a2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:41014/0 (socket says 172.21.15.121:41014) 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcae77fe700 1 -- 172.21.15.121:0/1346512111 learned_addr learned my addr 172.21.15.121:0/1346512111 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcae6ffd700 1 --2- 172.21.15.121:0/1346512111 >> v2:172.21.15.121:3300/0 conn(0x7fcae809f6a0 0x7fcae80a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.074+0000 7fcae7fff700 1 --2- 172.21.15.121:0/1346512111 >> v2:172.21.15.123:3300/0 conn(0x7fcae80a33e0 0x7fcae81475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae6ffd700 1 -- 172.21.15.121:0/1346512111 >> v2:172.21.15.123:3300/0 conn(0x7fcae80a33e0 msgr2=0x7fcae81475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae6ffd700 1 --2- 172.21.15.121:0/1346512111 >> v2:172.21.15.123:3300/0 conn(0x7fcae80a33e0 0x7fcae81475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae6ffd700 1 -- 172.21.15.121:0/1346512111 >> v1:172.21.15.121:6789/0 conn(0x7fcae8069600 legacy=0x7fcae80a2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae6ffd700 1 -- 172.21.15.121:0/1346512111 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae809fef0 con 0x7fcae809f6a0 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae67fc700 1 -- 172.21.15.121:0/1346512111 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027145612 0 0) 0x7fcae80699d0 con 0x7fcae8069600 2023-09-29T23:09:47.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae6ffd700 1 --2- 172.21.15.121:0/1346512111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae809f6a0 0x7fcae80a1370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcae80a5540 tx=0x7fcad80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfce5e8e0c41215 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae67fc700 1 -- 172.21.15.121:0/1346512111 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad800b6b0 con 0x7fcae809f6a0 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae67fc700 1 -- 172.21.15.121:0/1346512111 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad800b810 con 0x7fcae809f6a0 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcae67fc700 1 -- 172.21.15.121:0/1346512111 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad800bbb0 con 0x7fcae809f6a0 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 -- 172.21.15.121:0/1346512111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae809f6a0 msgr2=0x7fcae80a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 --2- 172.21.15.121:0/1346512111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae809f6a0 0x7fcae80a1370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcae80a5540 tx=0x7fcad80091a0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 -- 172.21.15.121:0/1346512111 shutdown_connections 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 --2- 172.21.15.121:0/1346512111 >> v2:172.21.15.123:3300/0 conn(0x7fcae80a33e0 0x7fcae81475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 --2- 172.21.15.121:0/1346512111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae809f6a0 0x7fcae80a1370 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 -- 172.21.15.121:0/1346512111 >> 172.21.15.121:0/1346512111 conn(0x7fcae8067440 msgr2=0x7fcae8067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 -- 172.21.15.121:0/1346512111 shutdown_connections 2023-09-29T23:09:47.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.075+0000 7fcaee163700 1 -- 172.21.15.121:0/1346512111 wait complete. 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.077+0000 7fcaee163700 1 Processor -- start 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.077+0000 7fcaee163700 1 -- start start 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcaee163700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae8069600 0x7fcae8144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcaee163700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcae809f6a0 0x7fcae8146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcaee163700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae80a33e0 0x7fcae8147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcaee163700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81463e0 con 0x7fcae80a33e0 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcaee163700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae8144d90 con 0x7fcae809f6a0 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcaee163700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae8140980 con 0x7fcae8069600 2023-09-29T23:09:47.081 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae8069600 0x7fcae8144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae80a33e0 0x7fcae8147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae8069600 0x7fcae8144430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44976/0 (socket says 172.21.15.121:44976) 2023-09-29T23:09:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae80a33e0 0x7fcae8147020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47492/0 (socket says 172.21.15.121:47492) 2023-09-29T23:09:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 -- 172.21.15.121:0/1133849019 learned_addr learned my addr 172.21.15.121:0/1133849019 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae6ffd700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcae809f6a0 0x7fcae8146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 -- 172.21.15.121:0/1133849019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcae809f6a0 msgr2=0x7fcae8146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:47.082 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcae809f6a0 0x7fcae8146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 -- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae80a33e0 msgr2=0x7fcae8147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae80a33e0 0x7fcae8147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 -- 172.21.15.121:0/1133849019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad8009420 con 0x7fcae8069600 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcae77fe700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae8069600 0x7fcae8144430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcad000c7e0 tx=0x7fcad0007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcadf7fe700 1 -- 172.21.15.121:0/1133849019 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0005b30 con 0x7fcae8069600 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcadf7fe700 1 -- 172.21.15.121:0/1133849019 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad0005c90 con 0x7fcae8069600 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcadf7fe700 1 -- 172.21.15.121:0/1133849019 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0007300 con 0x7fcae8069600 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.078+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae8143590 con 0x7fcae8069600 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.079+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae813af70 con 0x7fcae8069600 2023-09-29T23:09:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.079+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae8143780 con 0x7fcae8069600 2023-09-29T23:09:47.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.080+0000 7fcadf7fe700 1 -- 172.21.15.121:0/1133849019 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcad001b3b0 con 0x7fcae8069600 2023-09-29T23:09:47.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.080+0000 7fcadf7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:47.084 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.081+0000 7fcadf7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.081+0000 7fcadf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.081+0000 7fcadf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.081+0000 7fcadf7fe700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcac8098780 0x7fcac809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.081+0000 7fcae6ffd700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcac8098780 0x7fcac809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.081+0000 7fcadf7fe700 1 -- 172.21.15.121:0/1133849019 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fcad00bf950 con 0x7fcae8069600 2023-09-29T23:09:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.081+0000 7fcae6ffd700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcac8098780 0x7fcac809ac30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcad80093f0 tx=0x7fcad8017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.085+0000 7fcadf7fe700 1 -- 172.21.15.121:0/1133849019 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcad008cf60 con 0x7fcae8069600 2023-09-29T23:09:47.307 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.306+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcae8071660 con 0x7fcae8069600 2023-09-29T23:09:47.307 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.306+0000 7fcadf7fe700 1 -- 172.21.15.121:0/1133849019 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcad0017100 con 0x7fcae8069600 2023-09-29T23:09:47.308 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:47.308 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.308+0000 7fcaee163700 10 mgrc shutdown 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcac8098780 msgr2=0x7fcac809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcac8098780 0x7fcac809ac30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcad80093f0 tx=0x7fcad8017000 comp rx=0 tx=0).stop 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae8069600 msgr2=0x7fcae8144430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae8069600 0x7fcae8144430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcad000c7e0 tx=0x7fcad0007df0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 shutdown_connections 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fcac8098780 0x7fcac809ac30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcae80a33e0 0x7fcae8147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae8069600 0x7fcae8144430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.310 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 --2- 172.21.15.121:0/1133849019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcae809f6a0 0x7fcae8146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:47.311 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 >> 172.21.15.121:0/1133849019 conn(0x7fcae8067440 msgr2=0x7fcae8068980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:47.311 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 shutdown_connections 2023-09-29T23:09:47.311 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:47.309+0000 7fcaee163700 1 -- 172.21.15.121:0/1133849019 wait complete. 2023-09-29T23:09:50.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:50.450 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.450+0000 7f049a7da700 1 Processor -- start 2023-09-29T23:09:50.451 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f049a7da700 1 -- start start 2023-09-29T23:09:50.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f049a7da700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f04940a6950 0x7f04940a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:50.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f049a7da700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04941458e0 0x7f0494147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:50.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f049a7da700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04940a85e0 con 0x7f04940a73b0 2023-09-29T23:09:50.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f049a7da700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04940a6d20 con 0x7f04941458e0 2023-09-29T23:09:50.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f049a7da700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04940a91a0 con 0x7f04940a6950 2023-09-29T23:09:50.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f0498d77700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04941458e0 0x7f0494147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:50.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f0498d77700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04941458e0 0x7f0494147cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47508/0 (socket says 172.21.15.121:47508) 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f0498d77700 1 -- 172.21.15.121:0/3827799466 learned_addr learned my addr 172.21.15.121:0/3827799466 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.451+0000 7f0493fff700 1 --2- 172.21.15.121:0/3827799466 >> v2:172.21.15.123:3300/0 conn(0x7f04940a6950 0x7f04940a8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0498d77700 1 -- 172.21.15.121:0/3827799466 >> v2:172.21.15.123:3300/0 conn(0x7f04940a6950 msgr2=0x7f04940a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0498d77700 1 --2- 172.21.15.121:0/3827799466 >> v2:172.21.15.123:3300/0 conn(0x7f04940a6950 0x7f04940a8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0498d77700 1 -- 172.21.15.121:0/3827799466 >> v1:172.21.15.123:6789/0 conn(0x7f04940a73b0 legacy=0x7f049413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0498d77700 1 -- 172.21.15.121:0/3827799466 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04940a9440 con 0x7f04941458e0 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0498d77700 1 --2- 172.21.15.121:0/3827799466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494147cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f049406ca90 tx=0x7f048c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1ba827670e963e server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0492ffd700 1 -- 172.21.15.121:0/3827799466 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c00e070 con 0x7f04941458e0 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0492ffd700 1 -- 172.21.15.121:0/3827799466 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c00bae0 con 0x7f04941458e0 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f0492ffd700 1 -- 172.21.15.121:0/3827799466 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c0092e0 con 0x7f04941458e0 2023-09-29T23:09:50.454 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f049a7da700 1 -- 172.21.15.121:0/3827799466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 msgr2=0x7f0494147cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f049a7da700 1 --2- 172.21.15.121:0/3827799466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494147cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f049406ca90 tx=0x7f048c009840 comp rx=0 tx=0).stop 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f049a7da700 1 -- 172.21.15.121:0/3827799466 shutdown_connections 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f049a7da700 1 --2- 172.21.15.121:0/3827799466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494147cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f049a7da700 1 --2- 172.21.15.121:0/3827799466 >> v2:172.21.15.123:3300/0 conn(0x7f04940a6950 0x7f04940a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.452+0000 7f049a7da700 1 -- 172.21.15.121:0/3827799466 >> 172.21.15.121:0/3827799466 conn(0x7f049406c010 msgr2=0x7f049406ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.453+0000 7f049a7da700 1 -- 172.21.15.121:0/3827799466 shutdown_connections 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.453+0000 7f049a7da700 1 -- 172.21.15.121:0/3827799466 wait complete. 2023-09-29T23:09:50.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.453+0000 7f049a7da700 1 Processor -- start 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f049a7da700 1 -- start start 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f049a7da700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04940a6950 0x7f0494138f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f049a7da700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04940a73b0 0x7f04940b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f049a7da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494067440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f049a7da700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04940b2920 con 0x7f04941458e0 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f049a7da700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04940b2be0 con 0x7f04940a6950 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f049a7da700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04940b25f0 con 0x7f04940a73b0 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f0498d77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494067440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:50.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f04937fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04940a73b0 0x7f04940b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f0498d77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494067440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47524/0 (socket says 172.21.15.121:47524) 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f04937fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04940a73b0 0x7f04940b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44978/0 (socket says 172.21.15.121:44978) 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f0498d77700 1 -- 172.21.15.121:0/2446469000 learned_addr learned my addr 172.21.15.121:0/2446469000 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.454+0000 7f0493fff700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04940a6950 0x7f0494138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f0498d77700 1 -- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04940a73b0 msgr2=0x7f04940b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f0498d77700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04940a73b0 0x7f04940b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f0498d77700 1 -- 172.21.15.121:0/2446469000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04940a6950 msgr2=0x7f0494138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f0498d77700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04940a6950 0x7f0494138f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f0498d77700 1 -- 172.21.15.121:0/2446469000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c009ae0 con 0x7f04941458e0 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f04937fe700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04940a73b0 0x7f04940b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:50.459 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f0498d77700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494067440 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f04940a8770 tx=0x7f048c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f047bfff700 1 -- 172.21.15.121:0/2446469000 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c008d20 con 0x7f04941458e0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f047bfff700 1 -- 172.21.15.121:0/2446469000 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c008e80 con 0x7f04941458e0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049406d180 con 0x7f04941458e0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f047bfff700 1 -- 172.21.15.121:0/2446469000 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c01c660 con 0x7f04941458e0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.455+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0494143350 con 0x7f04941458e0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.456+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0494071660 con 0x7f04941458e0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.457+0000 7f047bfff700 1 -- 172.21.15.121:0/2446469000 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f048c003040 con 0x7f04941458e0 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.457+0000 7f047bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.457+0000 7f047bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.457+0000 7f047bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.457+0000 7f047bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:50.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.457+0000 7f047bfff700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f04800987b0 0x7f048009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:50.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.458+0000 7f0493fff700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f04800987b0 0x7f048009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:50.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.458+0000 7f047bfff700 1 -- 172.21.15.121:0/2446469000 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f048c0c20c0 con 0x7f04941458e0 2023-09-29T23:09:50.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.458+0000 7f0493fff700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f04800987b0 0x7f048009ac60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0484006fd0 tx=0x7f0484008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:50.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.461+0000 7f047bfff700 1 -- 172.21.15.121:0/2446469000 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f048c08f770 con 0x7f04941458e0 2023-09-29T23:09:50.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.681+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04940a9790 con 0x7f04941458e0 2023-09-29T23:09:50.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.681+0000 7f047bfff700 1 -- 172.21.15.121:0/2446469000 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f048c092d90 con 0x7f04941458e0 2023-09-29T23:09:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:50.681 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:50.683 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 10 mgrc shutdown 2023-09-29T23:09:50.683 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f04800987b0 msgr2=0x7f048009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f04800987b0 0x7f048009ac60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0484006fd0 tx=0x7f0484008040 comp rx=0 tx=0).stop 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 msgr2=0x7f0494067440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494067440 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f04940a8770 tx=0x7f048c018000 comp rx=0 tx=0).stop 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 shutdown_connections 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f04800987b0 0x7f048009ac60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04941458e0 0x7f0494067440 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04940a6950 0x7f0494138f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 --2- 172.21.15.121:0/2446469000 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04940a73b0 0x7f04940b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 >> 172.21.15.121:0/2446469000 conn(0x7f049406c010 msgr2=0x7f049413e710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 shutdown_connections 2023-09-29T23:09:50.684 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:50.683+0000 7f049a7da700 1 -- 172.21.15.121:0/2446469000 wait complete. 2023-09-29T23:09:53.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:53.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.825+0000 7fd292561700 1 Processor -- start 2023-09-29T23:09:53.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.825+0000 7fd292561700 1 -- start start 2023-09-29T23:09:53.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd292561700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c146ba0 0x7fd28c148f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:53.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd292561700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c141700 con 0x7fd28c0a6630 2023-09-29T23:09:53.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd292561700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c06ee60 con 0x7fd28c0a6a00 2023-09-29T23:09:53.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd292561700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0b4370 con 0x7fd28c146ba0 2023-09-29T23:09:53.827 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd290afe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c146ba0 0x7fd28c148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd28bfff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd28c0a6630 0x7fd28c140ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:41018/0 (socket says 172.21.15.121:41018) 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd28bfff700 1 -- 172.21.15.121:0/1797553458 learned_addr learned my addr 172.21.15.121:0/1797553458 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd28affd700 1 -- 172.21.15.121:0/1797553458 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924476441 0 0) 0x7fd28c141700 con 0x7fd28c0a6630 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd28affd700 1 -- 172.21.15.121:0/1797553458 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd274003570 con 0x7fd28c0a6630 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd290afe700 1 -- 172.21.15.121:0/1797553458 >> v1:172.21.15.123:6789/0 conn(0x7fd28c0a6a00 legacy=0x7fd28c0a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd290afe700 1 -- 172.21.15.121:0/1797553458 >> v1:172.21.15.121:6789/0 conn(0x7fd28c0a6630 legacy=0x7fd28c140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd290afe700 1 -- 172.21.15.121:0/1797553458 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c140980 con 0x7fd28c146ba0 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd290afe700 1 --2- 172.21.15.121:0/1797553458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c146ba0 0x7fd28c148f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd28c06e3d0 tx=0x7fd284009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce3bf8cc4da9f4a6 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd28affd700 1 -- 172.21.15.121:0/1797553458 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284016070 con 0x7fd28c146ba0 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd28affd700 1 -- 172.21.15.121:0/1797553458 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28400bb60 con 0x7fd28c146ba0 2023-09-29T23:09:53.828 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.826+0000 7fd28affd700 1 -- 172.21.15.121:0/1797553458 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28400e070 con 0x7fd28c146ba0 2023-09-29T23:09:53.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.827+0000 7fd292561700 1 -- 172.21.15.121:0/1797553458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c146ba0 msgr2=0x7fd28c148f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:53.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.827+0000 7fd292561700 1 --2- 172.21.15.121:0/1797553458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c146ba0 0x7fd28c148f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd28c06e3d0 tx=0x7fd284009780 comp rx=0 tx=0).stop 2023-09-29T23:09:53.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.827+0000 7fd292561700 1 -- 172.21.15.121:0/1797553458 shutdown_connections 2023-09-29T23:09:53.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.827+0000 7fd292561700 1 --2- 172.21.15.121:0/1797553458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c146ba0 0x7fd28c148f80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:53.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.827+0000 7fd292561700 1 -- 172.21.15.121:0/1797553458 >> 172.21.15.121:0/1797553458 conn(0x7fd28c06c010 msgr2=0x7fd28c06c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:53.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.827+0000 7fd292561700 1 -- 172.21.15.121:0/1797553458 shutdown_connections 2023-09-29T23:09:53.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.827+0000 7fd292561700 1 -- 172.21.15.121:0/1797553458 wait complete. 2023-09-29T23:09:53.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.828+0000 7fd292561700 1 Processor -- start 2023-09-29T23:09:53.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.828+0000 7fd292561700 1 -- start start 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c0a6630 0x7fd28c133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd28c0a6a00 0x7fd28c135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd28c146ba0 0x7fd28c1342e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c133b40 con 0x7fd28c146ba0 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c140480 con 0x7fd28c0a6a00 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c136520 con 0x7fd28c0a6630 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd28bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c0a6630 0x7fd28c133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd28c146ba0 0x7fd28c1342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd28c146ba0 0x7fd28c1342e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47532/0 (socket says 172.21.15.121:47532) 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd28bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c0a6630 0x7fd28c133600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44990/0 (socket says 172.21.15.121:44990) 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 -- 172.21.15.121:0/561039112 learned_addr learned my addr 172.21.15.121:0/561039112 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd28b7fe700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd28c0a6a00 0x7fd28c135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 -- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c0a6630 msgr2=0x7fd28c133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c0a6630 0x7fd28c133600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 -- 172.21.15.121:0/561039112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd28c0a6a00 msgr2=0x7fd28c135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd28c0a6a00 0x7fd28c135350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 -- 172.21.15.121:0/561039112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284009a20 con 0x7fd28c146ba0 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd28bfff700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c0a6630 0x7fd28c133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd290afe700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd28c146ba0 0x7fd28c1342e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd2840097c0 tx=0x7fd284018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd273fff700 1 -- 172.21.15.121:0/561039112 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284012dc0 con 0x7fd28c146ba0 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd273fff700 1 -- 172.21.15.121:0/561039112 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd284027470 con 0x7fd28c146ba0 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c1367e0 con 0x7fd28c146ba0 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd273fff700 1 -- 172.21.15.121:0/561039112 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284027830 con 0x7fd28c146ba0 2023-09-29T23:09:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.829+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c138290 con 0x7fd28c146ba0 2023-09-29T23:09:53.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.830+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c071660 con 0x7fd28c146ba0 2023-09-29T23:09:53.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.832+0000 7fd273fff700 1 -- 172.21.15.121:0/561039112 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd284024070 con 0x7fd28c146ba0 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.832+0000 7fd273fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.833+0000 7fd273fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.833+0000 7fd273fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.833+0000 7fd273fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.833+0000 7fd273fff700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd278098780 0x7fd27809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.833+0000 7fd28bfff700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd278098780 0x7fd27809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.833+0000 7fd273fff700 1 -- 172.21.15.121:0/561039112 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fd2840c3410 con 0x7fd28c146ba0 2023-09-29T23:09:53.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.833+0000 7fd28bfff700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd278098780 0x7fd27809ac30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd280007fd0 tx=0x7fd280009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:53.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:53.836+0000 7fd273fff700 1 -- 172.21.15.121:0/561039112 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd284090a20 con 0x7fd28c146ba0 2023-09-29T23:09:54.057 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.057+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd28c0a5b70 con 0x7fd28c146ba0 2023-09-29T23:09:54.057 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.057+0000 7fd273fff700 1 -- 172.21.15.121:0/561039112 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd284094040 con 0x7fd28c146ba0 2023-09-29T23:09:54.057 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:54.058 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:54.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 10 mgrc shutdown 2023-09-29T23:09:54.059 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd278098780 msgr2=0x7fd27809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd278098780 0x7fd27809ac30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd280007fd0 tx=0x7fd280009040 comp rx=0 tx=0).stop 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd28c146ba0 msgr2=0x7fd28c1342e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd28c146ba0 0x7fd28c1342e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd2840097c0 tx=0x7fd284018000 comp rx=0 tx=0).stop 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 shutdown_connections 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd278098780 0x7fd27809ac30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd28c146ba0 0x7fd28c1342e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd28c0a6630 0x7fd28c133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 --2- 172.21.15.121:0/561039112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd28c0a6a00 0x7fd28c135350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 >> 172.21.15.121:0/561039112 conn(0x7fd28c06c010 msgr2=0x7fd28c0a4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 shutdown_connections 2023-09-29T23:09:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:54.059+0000 7fd292561700 1 -- 172.21.15.121:0/561039112 wait complete. 2023-09-29T23:09:57.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.201+0000 7f9da9de4700 1 Processor -- start 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.201+0000 7f9da9de4700 1 -- start start 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da9de4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da40a53e0 0x7f9da4140b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da9de4700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9da41410a0 0x7f9da414ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da9de4700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da4141530 con 0x7f9da406ac70 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da9de4700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da406e110 con 0x7f9da41410a0 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da9de4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da406e3d0 con 0x7f9da40a53e0 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da2ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da40a53e0 0x7f9da4140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da2ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da40a53e0 0x7f9da4140b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:41830/0 (socket says 172.21.15.121:41830) 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da2ffd700 1 -- 172.21.15.121:0/3082792470 learned_addr learned my addr 172.21.15.121:0/3082792470 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da3fff700 1 --2- 172.21.15.121:0/3082792470 >> v2:172.21.15.121:3300/0 conn(0x7f9da41410a0 0x7f9da414ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da2ffd700 1 -- 172.21.15.121:0/3082792470 >> v2:172.21.15.121:3300/0 conn(0x7f9da41410a0 msgr2=0x7f9da414ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da2ffd700 1 --2- 172.21.15.121:0/3082792470 >> v2:172.21.15.121:3300/0 conn(0x7f9da41410a0 0x7f9da414ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.202+0000 7f9da2ffd700 1 -- 172.21.15.121:0/3082792470 >> v1:172.21.15.121:6789/0 conn(0x7f9da406ac70 legacy=0x7f9da406e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da27fc700 1 -- 172.21.15.121:0/3082792470 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976323016 0 0) 0x7f9da4141530 con 0x7f9da406ac70 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da2ffd700 1 -- 172.21.15.121:0/3082792470 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da4140580 con 0x7f9da40a53e0 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da3fff700 1 --2- 172.21.15.121:0/3082792470 >> v2:172.21.15.121:3300/0 conn(0x7f9da41410a0 0x7f9da414ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da2ffd700 1 --2- 172.21.15.121:0/3082792470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da40a53e0 0x7f9da4140b60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0099b0 tx=0x7f9d8c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57c1e66c16f73fd5 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da27fc700 1 -- 172.21.15.121:0/3082792470 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c009ea0 con 0x7f9da40a53e0 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da27fc700 1 -- 172.21.15.121:0/3082792470 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c00b860 con 0x7f9da40a53e0 2023-09-29T23:09:57.205 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da27fc700 1 -- 172.21.15.121:0/3082792470 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00bc00 con 0x7f9da40a53e0 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 -- 172.21.15.121:0/3082792470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da40a53e0 msgr2=0x7f9da4140b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 --2- 172.21.15.121:0/3082792470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da40a53e0 0x7f9da4140b60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0099b0 tx=0x7f9d8c009250 comp rx=0 tx=0).stop 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 -- 172.21.15.121:0/3082792470 shutdown_connections 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 --2- 172.21.15.121:0/3082792470 >> v2:172.21.15.121:3300/0 conn(0x7f9da41410a0 0x7f9da414ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 --2- 172.21.15.121:0/3082792470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da40a53e0 0x7f9da4140b60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 -- 172.21.15.121:0/3082792470 >> 172.21.15.121:0/3082792470 conn(0x7f9da4067440 msgr2=0x7f9da4067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 -- 172.21.15.121:0/3082792470 shutdown_connections 2023-09-29T23:09:57.206 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.203+0000 7f9da9de4700 1 -- 172.21.15.121:0/3082792470 wait complete. 2023-09-29T23:09:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 Processor -- start 2023-09-29T23:09:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 -- start start 2023-09-29T23:09:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9da406ac70 0x7f9da409be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da40a53e0 0x7f9da409cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da41410a0 0x7f9da413e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da409c870 con 0x7f9da40a53e0 2023-09-29T23:09:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da409a580 con 0x7f9da406ac70 2023-09-29T23:09:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da9de4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da40a2de0 con 0x7f9da41410a0 2023-09-29T23:09:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da3fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da41410a0 0x7f9da413e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da40a53e0 0x7f9da409cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da3fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da41410a0 0x7f9da413e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:41846/0 (socket says 172.21.15.121:41846) 2023-09-29T23:09:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da40a53e0 0x7f9da409cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45228/0 (socket says 172.21.15.121:45228) 2023-09-29T23:09:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da3fff700 1 -- 172.21.15.121:0/522082826 learned_addr learned my addr 172.21.15.121:0/522082826 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:09:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da37fe700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9da406ac70 0x7f9da409be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da2ffd700 1 -- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da41410a0 msgr2=0x7f9da413e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da2ffd700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da41410a0 0x7f9da413e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da2ffd700 1 -- 172.21.15.121:0/522082826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9da406ac70 msgr2=0x7f9da409be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da2ffd700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9da406ac70 0x7f9da409be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da2ffd700 1 -- 172.21.15.121:0/522082826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c009a20 con 0x7f9da40a53e0 2023-09-29T23:09:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.205+0000 7f9da3fff700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da41410a0 0x7f9da413e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:09:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.206+0000 7f9da37fe700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9da406ac70 0x7f9da409be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:09:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.206+0000 7f9da2ffd700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da40a53e0 0x7f9da409cef0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9da406cda0 tx=0x7f9d8c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.206+0000 7f9d9b7fe700 1 -- 172.21.15.121:0/522082826 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00bbe0 con 0x7f9da40a53e0 2023-09-29T23:09:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.206+0000 7f9d9b7fe700 1 -- 172.21.15.121:0/522082826 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c011420 con 0x7f9da40a53e0 2023-09-29T23:09:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.206+0000 7f9d9b7fe700 1 -- 172.21.15.121:0/522082826 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c011800 con 0x7f9da40a53e0 2023-09-29T23:09:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.206+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c0094f0 con 0x7f9da40a53e0 2023-09-29T23:09:57.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.206+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da40a35b0 con 0x7f9da40a53e0 2023-09-29T23:09:57.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.207+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da4071660 con 0x7f9da40a53e0 2023-09-29T23:09:57.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.208+0000 7f9d9b7fe700 1 -- 172.21.15.121:0/522082826 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9d8c011c30 con 0x7f9da40a53e0 2023-09-29T23:09:57.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.208+0000 7f9d9b7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:09:57.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.208+0000 7f9d9b7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:09:57.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.208+0000 7f9d9b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:57.214 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.208+0000 7f9d9b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:09:57.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.208+0000 7f9d9b7fe700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9d840987b0 0x7f9d8409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:09:57.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.209+0000 7f9da37fe700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9d840987b0 0x7f9d8409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:09:57.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.209+0000 7f9d9b7fe700 1 -- 172.21.15.121:0/522082826 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f9d8c0c1dd0 con 0x7f9da40a53e0 2023-09-29T23:09:57.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.209+0000 7f9da37fe700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9d840987b0 0x7f9d8409ac60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9d94001820 tx=0x7f9d94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:09:57.215 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.213+0000 7f9d9b7fe700 1 -- 172.21.15.121:0/522082826 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d8c08f480 con 0x7f9da40a53e0 2023-09-29T23:09:57.434 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.434+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9da40a6bd0 con 0x7f9da40a53e0 2023-09-29T23:09:57.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.435+0000 7f9d9b7fe700 1 -- 172.21.15.121:0/522082826 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9d8c015100 con 0x7f9da40a53e0 2023-09-29T23:09:57.435 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:09:57.435 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 10 mgrc shutdown 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9d840987b0 msgr2=0x7f9d8409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9d840987b0 0x7f9d8409ac60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9d94001820 tx=0x7f9d94009040 comp rx=0 tx=0).stop 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da40a53e0 msgr2=0x7f9da409cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da40a53e0 0x7f9da409cef0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9da406cda0 tx=0x7f9d8c017000 comp rx=0 tx=0).stop 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 shutdown_connections 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9d840987b0 0x7f9d8409ac60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9da41410a0 0x7f9da413e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9da406ac70 0x7f9da409be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 --2- 172.21.15.121:0/522082826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da40a53e0 0x7f9da409cef0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:09:57.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 >> 172.21.15.121:0/522082826 conn(0x7f9da4067440 msgr2=0x7f9da40b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:09:57.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 shutdown_connections 2023-09-29T23:09:57.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:09:57.437+0000 7f9da9de4700 1 -- 172.21.15.121:0/522082826 wait complete. 2023-09-29T23:09:59.999 INFO:tasks.ceph.mon.a.smithi121.stderr:2023-09-29T23:09:59.999+0000 7fd175b13700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:10:00.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 Processor -- start 2023-09-29T23:10:00.581 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 -- start start 2023-09-29T23:10:00.583 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f92981363c0 0x7f92980ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:00.583 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981438e0 0x7f92980b3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:00.583 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9298144040 0x7f9298067440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:00.583 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9298136790 con 0x7f9298144040 2023-09-29T23:10:00.583 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92980ac7f0 con 0x7f92981363c0 2023-09-29T23:10:00.583 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.581+0000 7f929f9ed700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92980a5470 con 0x7f92981438e0 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981438e0 0x7f92980b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929df8a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9298144040 0x7f9298067440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981438e0 0x7f92980b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:41856/0 (socket says 172.21.15.121:41856) 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 -- 172.21.15.121:0/2879309686 learned_addr learned my addr 172.21.15.121:0/2879309686 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929d789700 1 --2- 172.21.15.121:0/2879309686 >> v2:172.21.15.123:3300/0 conn(0x7f92981363c0 0x7f92980ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 -- 172.21.15.121:0/2879309686 >> v2:172.21.15.123:3300/0 conn(0x7f92981363c0 msgr2=0x7f92980ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 --2- 172.21.15.121:0/2879309686 >> v2:172.21.15.123:3300/0 conn(0x7f92981363c0 0x7f92980ac2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 -- 172.21.15.121:0/2879309686 >> v2:172.21.15.121:3300/0 conn(0x7f9298144040 msgr2=0x7f9298067440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 --2- 172.21.15.121:0/2879309686 >> v2:172.21.15.121:3300/0 conn(0x7f9298144040 0x7f9298067440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 -- 172.21.15.121:0/2879309686 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92980a5730 con 0x7f92981438e0 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929df8a700 1 --2- 172.21.15.121:0/2879309686 >> v2:172.21.15.121:3300/0 conn(0x7f9298144040 0x7f9298067440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f929cf88700 1 --2- 172.21.15.121:0/2879309686 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981438e0 0x7f92980b3980 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f92980b41e0 tx=0x7f9288009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c701aa2162de51 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f928ffff700 1 -- 172.21.15.121:0/2879309686 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288009e70 con 0x7f92981438e0 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f928ffff700 1 -- 172.21.15.121:0/2879309686 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928800b860 con 0x7f92981438e0 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.582+0000 7f928ffff700 1 -- 172.21.15.121:0/2879309686 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928800bc00 con 0x7f92981438e0 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 -- 172.21.15.121:0/2879309686 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981438e0 msgr2=0x7f92980b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 --2- 172.21.15.121:0/2879309686 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981438e0 0x7f92980b3980 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f92980b41e0 tx=0x7f9288009250 comp rx=0 tx=0).stop 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 -- 172.21.15.121:0/2879309686 shutdown_connections 2023-09-29T23:10:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 --2- 172.21.15.121:0/2879309686 >> v2:172.21.15.121:3300/0 conn(0x7f9298144040 0x7f9298067440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 --2- 172.21.15.121:0/2879309686 >> v2:172.21.15.123:3300/0 conn(0x7f92981363c0 0x7f92980ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 --2- 172.21.15.121:0/2879309686 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981438e0 0x7f92980b3980 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 -- 172.21.15.121:0/2879309686 >> 172.21.15.121:0/2879309686 conn(0x7f929806c010 msgr2=0x7f929806ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:00.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 -- 172.21.15.121:0/2879309686 shutdown_connections 2023-09-29T23:10:00.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.583+0000 7f929f9ed700 1 -- 172.21.15.121:0/2879309686 wait complete. 2023-09-29T23:10:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.584+0000 7f929f9ed700 1 Processor -- start 2023-09-29T23:10:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.584+0000 7f929f9ed700 1 -- start start 2023-09-29T23:10:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981363c0 0x7f9298138aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92981438e0 0x7f92980b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9298144040 0x7f9298150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92980b2830 con 0x7f9298144040 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298137c40 con 0x7f92981438e0 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9298139260 con 0x7f92981363c0 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981363c0 0x7f9298138aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981363c0 0x7f9298138aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:41868/0 (socket says 172.21.15.121:41868) 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 -- 172.21.15.121:0/3650878327 learned_addr learned my addr 172.21.15.121:0/3650878327 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929df8a700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9298144040 0x7f9298150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929cf88700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92981438e0 0x7f92980b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 -- 172.21.15.121:0/3650878327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92981438e0 msgr2=0x7f92980b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92981438e0 0x7f92980b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 -- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9298144040 msgr2=0x7f9298150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9298144040 0x7f9298150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 -- 172.21.15.121:0/3650878327 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9290009a20 con 0x7f92981363c0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929df8a700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9298144040 0x7f9298150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929d789700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981363c0 0x7f9298138aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f929400d890 tx=0x7f9294008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f928d7fa700 1 -- 172.21.15.121:0/3650878327 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92940125f0 con 0x7f92981363c0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f928d7fa700 1 -- 172.21.15.121:0/3650878327 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9294012750 con 0x7f92981363c0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f928d7fa700 1 -- 172.21.15.121:0/3650878327 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294012ad0 con 0x7f92981363c0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92880094f0 con 0x7f92981363c0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.585+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92980b3150 con 0x7f92981363c0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.587+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92981390b0 con 0x7f92981363c0 2023-09-29T23:10:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.587+0000 7f928d7fa700 1 -- 172.21.15.121:0/3650878327 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f929401e460 con 0x7f92981363c0 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.587+0000 7f928d7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.588+0000 7f928d7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.588+0000 7f928d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.588+0000 7f928d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.588+0000 7f928d7fa700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f927c098780 0x7f927c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.588+0000 7f929cf88700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f927c098780 0x7f927c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.588+0000 7f928d7fa700 1 -- 172.21.15.121:0/3650878327 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f92940c09d0 con 0x7f92981363c0 2023-09-29T23:10:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.588+0000 7f929cf88700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f927c098780 0x7f927c09ac30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f92980b41e0 tx=0x7f9288017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:00.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.592+0000 7f928d7fa700 1 -- 172.21.15.121:0/3650878327 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f929408e080 con 0x7f92981363c0 2023-09-29T23:10:00.811 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.812+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9298071660 con 0x7f92981363c0 2023-09-29T23:10:00.812 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.812+0000 7f928d7fa700 1 -- 172.21.15.121:0/3650878327 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f92940916a0 con 0x7f92981363c0 2023-09-29T23:10:00.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:00.812 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:00.814 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 10 mgrc shutdown 2023-09-29T23:10:00.814 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f927c098780 msgr2=0x7f927c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:00.814 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f927c098780 0x7f927c09ac30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f92980b41e0 tx=0x7f9288017000 comp rx=0 tx=0).stop 2023-09-29T23:10:00.814 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981363c0 msgr2=0x7f9298138aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981363c0 0x7f9298138aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f929400d890 tx=0x7f9294008040 comp rx=0 tx=0).stop 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 shutdown_connections 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f927c098780 0x7f927c09ac30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9298144040 0x7f9298150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92981363c0 0x7f9298138aa0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 --2- 172.21.15.121:0/3650878327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92981438e0 0x7f92980b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 >> 172.21.15.121:0/3650878327 conn(0x7f929806c010 msgr2=0x7f929806ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 shutdown_connections 2023-09-29T23:10:00.815 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:00.814+0000 7f929f9ed700 1 -- 172.21.15.121:0/3650878327 wait complete. 2023-09-29T23:10:03.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:03.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.955+0000 7f047c033700 1 Processor -- start 2023-09-29T23:10:03.955 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.955+0000 7f047c033700 1 -- start start 2023-09-29T23:10:03.957 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f047c033700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f04740b3cf0 0x7f047414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f047c033700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0474141810 con 0x7f047409b770 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f047c033700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047409a7f0 con 0x7f0474141440 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f047c033700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04740a4df0 con 0x7f04740b3cf0 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f04795ce700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f047409b770 0x7f04740630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:44410/0 (socket says 172.21.15.121:44410) 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f04795ce700 1 -- 172.21.15.121:0/730657794 learned_addr learned my addr 172.21.15.121:0/730657794 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f047a5d0700 1 --2- 172.21.15.121:0/730657794 >> v2:172.21.15.123:3300/0 conn(0x7f04740b3cf0 0x7f047414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868403140 0 0) 0x7f0474141810 con 0x7f047409b770 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.956+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0460003570 con 0x7f047409b770 2023-09-29T23:10:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382643048 0 0) 0x7f047409a7f0 con 0x7f0474141440 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0474141810 con 0x7f0474141440 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2614354995 0 0) 0x7f0460003570 con 0x7f047409b770 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f047a5d0700 1 -- 172.21.15.121:0/730657794 >> v1:172.21.15.123:6789/0 conn(0x7f0474141440 legacy=0x7f04740a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f047a5d0700 1 -- 172.21.15.121:0/730657794 >> v1:172.21.15.121:6789/0 conn(0x7f047409b770 legacy=0x7f04740630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f047a5d0700 1 -- 172.21.15.121:0/730657794 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04740a50b0 con 0x7f04740b3cf0 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f047a5d0700 1 --2- 172.21.15.121:0/730657794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04740b3cf0 0x7f047414c1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f046c009e60 tx=0x7f046c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d14fe316626975b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c00e070 con 0x7f04740b3cf0 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c00b7a0 con 0x7f04740b3cf0 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.957+0000 7f0478dcd700 1 -- 172.21.15.121:0/730657794 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470001470 con 0x7f04740b3cf0 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.958+0000 7f047c033700 1 -- 172.21.15.121:0/730657794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04740b3cf0 msgr2=0x7f047414c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.958+0000 7f047c033700 1 --2- 172.21.15.121:0/730657794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04740b3cf0 0x7f047414c1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f046c009e60 tx=0x7f046c009780 comp rx=0 tx=0).stop 2023-09-29T23:10:03.959 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.958+0000 7f047c033700 1 -- 172.21.15.121:0/730657794 shutdown_connections 2023-09-29T23:10:03.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.958+0000 7f047c033700 1 --2- 172.21.15.121:0/730657794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04740b3cf0 0x7f047414c1a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:03.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.958+0000 7f047c033700 1 -- 172.21.15.121:0/730657794 >> 172.21.15.121:0/730657794 conn(0x7f0474069e90 msgr2=0x7f04740b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:03.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.958+0000 7f047c033700 1 -- 172.21.15.121:0/730657794 shutdown_connections 2023-09-29T23:10:03.960 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.958+0000 7f047c033700 1 -- 172.21.15.121:0/730657794 wait complete. 2023-09-29T23:10:03.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.959+0000 7f047c033700 1 Processor -- start 2023-09-29T23:10:03.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.959+0000 7f047c033700 1 -- start start 2023-09-29T23:10:03.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047c033700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f047409b770 0x7f0474145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:03.962 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047c033700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04740b3cf0 0x7f047409dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047c033700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0474141440 0x7f047409e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047c033700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047406e160 con 0x7f04740b3cf0 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047c033700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047406e360 con 0x7f0474141440 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047c033700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0474144c00 con 0x7f047409b770 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f047409b770 0x7f0474145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f04795ce700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04740b3cf0 0x7f047409dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f047409b770 0x7f0474145220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:41878/0 (socket says 172.21.15.121:41878) 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 -- 172.21.15.121:0/3594198658 learned_addr learned my addr 172.21.15.121:0/3594198658 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047a5d0700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0474141440 0x7f047409e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 -- 172.21.15.121:0/3594198658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0474141440 msgr2=0x7f047409e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:03.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0474141440 0x7f047409e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 -- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04740b3cf0 msgr2=0x7f047409dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04740b3cf0 0x7f047409dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 -- 172.21.15.121:0/3594198658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c009a20 con 0x7f047409b770 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f04795ce700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04740b3cf0 0x7f047409dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0479dcf700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f047409b770 0x7f0474145220 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f046400c5c0 tx=0x7f0464007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0469ffb700 1 -- 172.21.15.121:0/3594198658 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464010510 con 0x7f047409b770 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0469ffb700 1 -- 172.21.15.121:0/3594198658 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0464010670 con 0x7f047409b770 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047409e830 con 0x7f047409b770 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.960+0000 7f0469ffb700 1 -- 172.21.15.121:0/3594198658 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464010a30 con 0x7f047409b770 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.961+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047409f820 con 0x7f047409b770 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.961+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047406ab60 con 0x7f047409b770 2023-09-29T23:10:03.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f0469ffb700 1 -- 172.21.15.121:0/3594198658 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f0464002050 con 0x7f047409b770 2023-09-29T23:10:03.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f0469ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:03.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f0469ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:03.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f0469ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:03.965 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f0469ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:03.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f0469ffb700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0458098780 0x7f045809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:03.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f04795ce700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0458098780 0x7f045809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:03.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f0469ffb700 1 -- 172.21.15.121:0/3594198658 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f04640c08d0 con 0x7f047409b770 2023-09-29T23:10:03.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.963+0000 7f04795ce700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0458098780 0x7f045809ac30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f047409fa20 tx=0x7f0470009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:03.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:03.966+0000 7f0469ffb700 1 -- 172.21.15.121:0/3594198658 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f046408de50 con 0x7f047409b770 2023-09-29T23:10:04.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.187+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0474071660 con 0x7f047409b770 2023-09-29T23:10:04.187 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.187+0000 7f0469ffb700 1 -- 172.21.15.121:0/3594198658 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0464091470 con 0x7f047409b770 2023-09-29T23:10:04.187 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:04.187 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:04.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 10 mgrc shutdown 2023-09-29T23:10:04.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0458098780 msgr2=0x7f045809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:04.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0458098780 0x7f045809ac30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f047409fa20 tx=0x7f0470009040 comp rx=0 tx=0).stop 2023-09-29T23:10:04.189 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f047409b770 msgr2=0x7f0474145220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f047409b770 0x7f0474145220 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f046400c5c0 tx=0x7f0464007e00 comp rx=0 tx=0).stop 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 shutdown_connections 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f0458098780 0x7f045809ac30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0474141440 0x7f047409e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f047409b770 0x7f0474145220 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 --2- 172.21.15.121:0/3594198658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04740b3cf0 0x7f047409dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 >> 172.21.15.121:0/3594198658 conn(0x7f0474069e90 msgr2=0x7f047413efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 shutdown_connections 2023-09-29T23:10:04.190 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:04.189+0000 7f047c033700 1 -- 172.21.15.121:0/3594198658 wait complete. 2023-09-29T23:10:07.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:07.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.331+0000 7fb4d21ff700 1 Processor -- start 2023-09-29T23:10:07.331 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.331+0000 7fb4d21ff700 1 -- start start 2023-09-29T23:10:07.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4d21ff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc0b5020 0x7fb4cc09e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:07.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4d21ff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb4cc13ede0 0x7fb4cc06b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:07.333 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4d21ff700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0b53f0 con 0x7fb4cc06b470 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4d21ff700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc13e2e0 con 0x7fb4cc13ede0 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4d21ff700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc06ae80 con 0x7fb4cc0b5020 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc0b5020 0x7fb4cc09e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc0b5020 0x7fb4cc09e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:39718/0 (socket says 172.21.15.121:39718) 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2525862495 learned_addr learned my addr 172.21.15.121:0/2525862495 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cbfff700 1 --2- 172.21.15.121:0/2525862495 >> v2:172.21.15.123:3300/0 conn(0x7fb4cc13ede0 0x7fb4cc06b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2525862495 >> v2:172.21.15.123:3300/0 conn(0x7fb4cc13ede0 msgr2=0x7fb4cc06b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:07.334 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 --2- 172.21.15.121:0/2525862495 >> v2:172.21.15.123:3300/0 conn(0x7fb4cc13ede0 0x7fb4cc06b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2525862495 >> v1:172.21.15.123:6789/0 conn(0x7fb4cc06b470 legacy=0x7fb4cc13e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2525862495 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc06f390 con 0x7fb4cc0b5020 2023-09-29T23:10:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4cb7fe700 1 --2- 172.21.15.121:0/2525862495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc0b5020 0x7fb4cc09e260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00099b0 tx=0x7fb4c0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8fd7917ce3f65f2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4ca7fc700 1 -- 172.21.15.121:0/2525862495 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000b820 con 0x7fb4cc0b5020 2023-09-29T23:10:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4ca7fc700 1 -- 172.21.15.121:0/2525862495 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c000b980 con 0x7fb4cc0b5020 2023-09-29T23:10:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.332+0000 7fb4ca7fc700 1 -- 172.21.15.121:0/2525862495 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000bd20 con 0x7fb4cc0b5020 2023-09-29T23:10:07.335 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2525862495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc0b5020 msgr2=0x7fb4cc09e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:07.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2525862495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc0b5020 0x7fb4cc09e260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00099b0 tx=0x7fb4c0009250 comp rx=0 tx=0).stop 2023-09-29T23:10:07.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2525862495 shutdown_connections 2023-09-29T23:10:07.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2525862495 >> v2:172.21.15.123:3300/0 conn(0x7fb4cc13ede0 0x7fb4cc06b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2525862495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc0b5020 0x7fb4cc09e260 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2525862495 >> 172.21.15.121:0/2525862495 conn(0x7fb4cc06d0b0 msgr2=0x7fb4cc06d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:07.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2525862495 shutdown_connections 2023-09-29T23:10:07.336 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.333+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2525862495 wait complete. 2023-09-29T23:10:07.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.334+0000 7fb4d21ff700 1 Processor -- start 2023-09-29T23:10:07.339 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.334+0000 7fb4d21ff700 1 -- start start 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4d21ff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc06b470 0x7fb4cc09ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4d21ff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4cc0b5020 0x7fb4cc069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4d21ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4cc13ede0 0x7fb4cc06a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4d21ff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc09b580 con 0x7fb4cc13ede0 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4d21ff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc1437d0 con 0x7fb4cc0b5020 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4d21ff700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc1407a0 con 0x7fb4cc06b470 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc06b470 0x7fb4cc09ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:07.340 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc06b470 0x7fb4cc09ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:39732/0 (socket says 172.21.15.121:39732) 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2288333349 learned_addr learned my addr 172.21.15.121:0/2288333349 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4cc13ede0 0x7fb4cc06a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4caffd700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4cc0b5020 0x7fb4cc069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2288333349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4cc0b5020 msgr2=0x7fb4cc069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4cc0b5020 0x7fb4cc069c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4cc13ede0 msgr2=0x7fb4cc06a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4cc13ede0 0x7fb4cc06a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 -- 172.21.15.121:0/2288333349 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00094f0 con 0x7fb4cc06b470 2023-09-29T23:10:07.341 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4cb7fe700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc06b470 0x7fb4cc09ac70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009290 tx=0x7fb4c0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4b37fe700 1 -- 172.21.15.121:0/2288333349 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0011c30 con 0x7fb4cc06b470 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4b37fe700 1 -- 172.21.15.121:0/2288333349 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c0020070 con 0x7fb4cc06b470 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc099e90 con 0x7fb4cc06b470 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.335+0000 7fb4b37fe700 1 -- 172.21.15.121:0/2288333349 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c001b450 con 0x7fb4cc06b470 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.336+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc1469b0 con 0x7fb4cc06b470 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.336+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc09f960 con 0x7fb4cc06b470 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.338+0000 7fb4b37fe700 1 -- 172.21.15.121:0/2288333349 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb4c0022030 con 0x7fb4cc06b470 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.338+0000 7fb4b37fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:07.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.338+0000 7fb4b37fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:07.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.338+0000 7fb4b37fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:07.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.338+0000 7fb4b37fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:07.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.338+0000 7fb4b37fe700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb4b8098780 0x7fb4b809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:07.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.339+0000 7fb4caffd700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb4b8098780 0x7fb4b809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:07.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.339+0000 7fb4b37fe700 1 -- 172.21.15.121:0/2288333349 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fb4c00c2670 con 0x7fb4cc06b470 2023-09-29T23:10:07.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.339+0000 7fb4caffd700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb4b8098780 0x7fb4b809ac30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc0a15e0 tx=0x7fb4bc007ea0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:07.343 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.342+0000 7fb4b37fe700 1 -- 172.21.15.121:0/2288333349 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4c008fd20 con 0x7fb4cc06b470 2023-09-29T23:10:07.563 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.563+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb4cc09f960 con 0x7fb4cc06b470 2023-09-29T23:10:07.564 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.564+0000 7fb4b37fe700 1 -- 172.21.15.121:0/2288333349 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb4c0023e60 con 0x7fb4cc06b470 2023-09-29T23:10:07.564 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:07.564 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.565+0000 7fb4d21ff700 10 mgrc shutdown 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb4b8098780 msgr2=0x7fb4b809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb4b8098780 0x7fb4b809ac30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc0a15e0 tx=0x7fb4bc007ea0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc06b470 msgr2=0x7fb4cc09ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc06b470 0x7fb4cc09ac70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009290 tx=0x7fb4c0016000 comp rx=0 tx=0).stop 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 shutdown_connections 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb4b8098780 0x7fb4b809ac30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4cc13ede0 0x7fb4cc06a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.566 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4cc06b470 0x7fb4cc09ac70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 --2- 172.21.15.121:0/2288333349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4cc0b5020 0x7fb4cc069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:07.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 >> 172.21.15.121:0/2288333349 conn(0x7fb4cc06d0b0 msgr2=0x7fb4cc13e490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:07.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 shutdown_connections 2023-09-29T23:10:07.567 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:07.566+0000 7fb4d21ff700 1 -- 172.21.15.121:0/2288333349 wait complete. 2023-09-29T23:10:10.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:10.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.707+0000 7f632b1b6700 1 Processor -- start 2023-09-29T23:10:10.707 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.707+0000 7f632b1b6700 1 -- start start 2023-09-29T23:10:10.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f632b1b6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f63240630e0 0x7f632409c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:10.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f632b1b6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632409e600 con 0x7f63240634b0 2023-09-29T23:10:10.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f632b1b6700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63240a2d20 con 0x7f632409f580 2023-09-29T23:10:10.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f632b1b6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632409d8d0 con 0x7f63240630e0 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6328f52700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f63240630e0 0x7f632409c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6328f52700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f63240630e0 0x7f632409c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38724/0 (socket says 172.21.15.121:38724) 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6323fff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f63240634b0 0x7f632409c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33844/0 (socket says 172.21.15.121:33844) 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6328f52700 1 -- 172.21.15.121:0/2153682234 learned_addr learned my addr 172.21.15.121:0/2153682234 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6328f52700 1 -- 172.21.15.121:0/2153682234 >> v1:172.21.15.123:6789/0 conn(0x7f632409f580 legacy=0x7f632409ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6328f52700 1 -- 172.21.15.121:0/2153682234 >> v1:172.21.15.121:6789/0 conn(0x7f63240634b0 legacy=0x7f632409c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6328f52700 1 -- 172.21.15.121:0/2153682234 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632409db70 con 0x7f63240630e0 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f63237fe700 1 -- 172.21.15.121:0/2153682234 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809952756 0 0) 0x7f632409e600 con 0x7f63240634b0 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f6328f52700 1 --2- 172.21.15.121:0/2153682234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240630e0 0x7f632409c070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6318009990 tx=0x7f6318009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6c2ce4be40f59c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:10.710 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f63237fe700 1 -- 172.21.15.121:0/2153682234 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318005c10 con 0x7f63240630e0 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f63237fe700 1 -- 172.21.15.121:0/2153682234 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6318005d70 con 0x7f63240630e0 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.708+0000 7f63237fe700 1 -- 172.21.15.121:0/2153682234 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631800b840 con 0x7f63240630e0 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.709+0000 7f632b1b6700 1 -- 172.21.15.121:0/2153682234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240630e0 msgr2=0x7f632409c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.709+0000 7f632b1b6700 1 --2- 172.21.15.121:0/2153682234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240630e0 0x7f632409c070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6318009990 tx=0x7f6318009220 comp rx=0 tx=0).stop 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.709+0000 7f632b1b6700 1 -- 172.21.15.121:0/2153682234 shutdown_connections 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.709+0000 7f632b1b6700 1 --2- 172.21.15.121:0/2153682234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240630e0 0x7f632409c070 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.709+0000 7f632b1b6700 1 -- 172.21.15.121:0/2153682234 >> 172.21.15.121:0/2153682234 conn(0x7f6324069e90 msgr2=0x7f63240b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.709+0000 7f632b1b6700 1 -- 172.21.15.121:0/2153682234 shutdown_connections 2023-09-29T23:10:10.711 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.709+0000 7f632b1b6700 1 -- 172.21.15.121:0/2153682234 wait complete. 2023-09-29T23:10:10.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.710+0000 7f632b1b6700 1 Processor -- start 2023-09-29T23:10:10.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.710+0000 7f632b1b6700 1 -- start start 2023-09-29T23:10:10.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f632b1b6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63240630e0 0x7f63240972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:10.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f632b1b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240634b0 0x7f632406e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:10.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f632b1b6700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f632409f580 0x7f632413f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f632b1b6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324097800 con 0x7f63240634b0 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f632b1b6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632409a0a0 con 0x7f63240630e0 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f632b1b6700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6324097a20 con 0x7f632409f580 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240634b0 0x7f632406e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6329753700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f632409f580 0x7f632413f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6328f52700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63240630e0 0x7f63240972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240634b0 0x7f632406e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38726/0 (socket says 172.21.15.121:38726) 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 -- 172.21.15.121:0/3755550414 learned_addr learned my addr 172.21.15.121:0/3755550414 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 -- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f632409f580 msgr2=0x7f632413f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f632409f580 0x7f632413f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 -- 172.21.15.121:0/3755550414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63240630e0 msgr2=0x7f63240972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63240630e0 0x7f63240972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 -- 172.21.15.121:0/3755550414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6318009480 con 0x7f63240634b0 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6323fff700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240634b0 0x7f632406e8e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f631000c7e0 tx=0x7f6310007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6320ff9700 1 -- 172.21.15.121:0/3755550414 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310005a70 con 0x7f63240634b0 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f6320ff9700 1 -- 172.21.15.121:0/3755550414 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6310005bd0 con 0x7f63240634b0 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.711+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6324097d40 con 0x7f63240634b0 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.712+0000 7f6320ff9700 1 -- 172.21.15.121:0/3755550414 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63100070d0 con 0x7f63240634b0 2023-09-29T23:10:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.712+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632409a550 con 0x7f63240634b0 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.712+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632406ab60 con 0x7f63240634b0 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.713+0000 7f6320ff9700 1 -- 172.21.15.121:0/3755550414 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6310007440 con 0x7f63240634b0 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6320ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6320ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6320ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6320ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6320ff9700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6308098780 0x7f630809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6320ff9700 1 -- 172.21.15.121:0/3755550414 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f63100bfde0 con 0x7f63240634b0 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6328f52700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6308098780 0x7f630809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:10.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.714+0000 7f6328f52700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6308098780 0x7f630809ac30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6318009260 tx=0x7f6318015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:10.718 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.717+0000 7f6320ff9700 1 -- 172.21.15.121:0/3755550414 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f631008d400 con 0x7f63240634b0 2023-09-29T23:10:10.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.939+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6324071660 con 0x7f63240634b0 2023-09-29T23:10:10.939 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.939+0000 7f6320ff9700 1 -- 172.21.15.121:0/3755550414 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6310017020 con 0x7f63240634b0 2023-09-29T23:10:10.939 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:10.940 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.941+0000 7f632b1b6700 10 mgrc shutdown 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.941+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6308098780 msgr2=0x7f630809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.941+0000 7f632b1b6700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6308098780 0x7f630809ac30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6318009260 tx=0x7f6318015000 comp rx=0 tx=0).stop 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.941+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240634b0 msgr2=0x7f632406e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240634b0 0x7f632406e8e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f631000c7e0 tx=0x7f6310007e10 comp rx=0 tx=0).stop 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 shutdown_connections 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6308098780 0x7f630809ac30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f632409f580 0x7f632413f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63240630e0 0x7f63240972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:10.942 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 --2- 172.21.15.121:0/3755550414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63240634b0 0x7f632406e8e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:10.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 >> 172.21.15.121:0/3755550414 conn(0x7f6324069e90 msgr2=0x7f632409ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:10.943 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 shutdown_connections 2023-09-29T23:10:10.945 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:10.942+0000 7f632b1b6700 1 -- 172.21.15.121:0/3755550414 wait complete. 2023-09-29T23:10:13.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:14.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.083+0000 7f6cbf3c5700 1 Processor -- start 2023-09-29T23:10:14.083 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.083+0000 7f6cbf3c5700 1 -- start start 2023-09-29T23:10:14.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbf3c5700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6cb80a6740 0x7f6cb806e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:14.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbf3c5700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb806ed50 con 0x7f6cb806a5a0 2023-09-29T23:10:14.085 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbf3c5700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb806f010 con 0x7f6cb806d580 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbf3c5700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb806a970 con 0x7f6cb80a6740 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbc960700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6cb80a6740 0x7f6cb806e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbd161700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6cb806a5a0 0x7f6cb813d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33846/0 (socket says 172.21.15.121:33846) 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbd161700 1 -- 172.21.15.121:0/4184705190 learned_addr learned my addr 172.21.15.121:0/4184705190 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbc960700 1 -- 172.21.15.121:0/4184705190 >> v1:172.21.15.123:6789/0 conn(0x7f6cb806d580 legacy=0x7f6cb8147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cbc960700 1 -- 172.21.15.121:0/4184705190 >> v1:172.21.15.121:6789/0 conn(0x7f6cb806a5a0 legacy=0x7f6cb813d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbc960700 1 -- 172.21.15.121:0/4184705190 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb80a5220 con 0x7f6cb80a6740 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.084+0000 7f6cabfff700 1 -- 172.21.15.121:0/4184705190 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290009293 0 0) 0x7f6cb806ed50 con 0x7f6cb806a5a0 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbc960700 1 --2- 172.21.15.121:0/4184705190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb80a6740 0x7f6cb806e720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0098e0 tx=0x7f6cac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c1c0e403b7ce85 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cabfff700 1 -- 172.21.15.121:0/4184705190 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac009e80 con 0x7f6cb80a6740 2023-09-29T23:10:14.086 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cabfff700 1 -- 172.21.15.121:0/4184705190 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cac00b860 con 0x7f6cb80a6740 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cabfff700 1 -- 172.21.15.121:0/4184705190 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac00bbe0 con 0x7f6cb80a6740 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/4184705190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb80a6740 msgr2=0x7f6cb806e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/4184705190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb80a6740 0x7f6cb806e720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0098e0 tx=0x7f6cac0091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/4184705190 shutdown_connections 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/4184705190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb80a6740 0x7f6cb806e720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/4184705190 >> 172.21.15.121:0/4184705190 conn(0x7f6cb8068fd0 msgr2=0x7f6cb8069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/4184705190 shutdown_connections 2023-09-29T23:10:14.087 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.085+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/4184705190 wait complete. 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.086+0000 7f6cbf3c5700 1 Processor -- start 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.086+0000 7f6cbf3c5700 1 -- start start 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbf3c5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb806a5a0 0x7f6cb813e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbf3c5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb806d580 0x7f6cb8096d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbf3c5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cb80a6740 0x7f6cb8139b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbf3c5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb8097290 con 0x7f6cb806a5a0 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbf3c5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb8096900 con 0x7f6cb806d580 2023-09-29T23:10:14.090 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbf3c5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb813e9f0 con 0x7f6cb80a6740 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbd161700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb806a5a0 0x7f6cb813e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbd161700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb806a5a0 0x7f6cb813e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38748/0 (socket says 172.21.15.121:38748) 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbd161700 1 -- 172.21.15.121:0/2573821020 learned_addr learned my addr 172.21.15.121:0/2573821020 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbd962700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cb80a6740 0x7f6cb8139b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbc960700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb806d580 0x7f6cb8096d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.087+0000 7f6cbd161700 1 -- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cb80a6740 msgr2=0x7f6cb8139b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6cbd161700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cb80a6740 0x7f6cb8139b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6cbd161700 1 -- 172.21.15.121:0/2573821020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb806d580 msgr2=0x7f6cb8096d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6cbd161700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb806d580 0x7f6cb8096d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6cbd161700 1 -- 172.21.15.121:0/2573821020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac009420 con 0x7f6cb806a5a0 2023-09-29T23:10:14.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6cbd161700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb806a5a0 0x7f6cb813e380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6cb400c810 tx=0x7f6cb4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6ca97fa700 1 -- 172.21.15.121:0/2573821020 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4005c00 con 0x7f6cb806a5a0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6ca97fa700 1 -- 172.21.15.121:0/2573821020 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb4005d60 con 0x7f6cb806a5a0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb813ecb0 con 0x7f6cb806a5a0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6ca97fa700 1 -- 172.21.15.121:0/2573821020 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4007320 con 0x7f6cb806a5a0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.088+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb8095a80 con 0x7f6cb806a5a0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.089+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb8071660 con 0x7f6cb806a5a0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6ca97fa700 1 -- 172.21.15.121:0/2573821020 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6cb401b3b0 con 0x7f6cb806a5a0 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6ca97fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6ca97fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:14.092 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6ca97fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:14.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6ca97fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:14.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6ca97fa700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c9c0987b0 0x7f6c9c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:14.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6ca97fa700 1 -- 172.21.15.121:0/2573821020 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f6cb40bfc90 con 0x7f6cb806a5a0 2023-09-29T23:10:14.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.090+0000 7f6cbc960700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c9c0987b0 0x7f6c9c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:14.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.091+0000 7f6cbc960700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c9c0987b0 0x7f6c9c09ac60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0093f0 tx=0x7f6cac017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:14.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.094+0000 7f6ca97fa700 1 -- 172.21.15.121:0/2573821020 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6cb408d340 con 0x7f6cb806a5a0 2023-09-29T23:10:14.317 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.317+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6cb806baf0 con 0x7f6cb806a5a0 2023-09-29T23:10:14.317 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.317+0000 7f6ca97fa700 1 -- 172.21.15.121:0/2573821020 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6cb4017100 con 0x7f6cb806a5a0 2023-09-29T23:10:14.317 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:14.317 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:14.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 10 mgrc shutdown 2023-09-29T23:10:14.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c9c0987b0 msgr2=0x7f6c9c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:14.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c9c0987b0 0x7f6c9c09ac60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0093f0 tx=0x7f6cac017000 comp rx=0 tx=0).stop 2023-09-29T23:10:14.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb806a5a0 msgr2=0x7f6cb813e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:14.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb806a5a0 0x7f6cb813e380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6cb400c810 tx=0x7f6cb4007d90 comp rx=0 tx=0).stop 2023-09-29T23:10:14.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 shutdown_connections 2023-09-29T23:10:14.319 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f6c9c0987b0 0x7f6c9c09ac60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cb80a6740 0x7f6cb8139b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb806a5a0 0x7f6cb813e380 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 --2- 172.21.15.121:0/2573821020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb806d580 0x7f6cb8096d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:14.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 >> 172.21.15.121:0/2573821020 conn(0x7f6cb8068fd0 msgr2=0x7f6cb809d310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:14.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 shutdown_connections 2023-09-29T23:10:14.320 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:14.319+0000 7f6cbf3c5700 1 -- 172.21.15.121:0/2573821020 wait complete. 2023-09-29T23:10:17.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.461+0000 7f3a0b499700 1 Processor -- start 2023-09-29T23:10:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.461+0000 7f3a0b499700 1 -- start start 2023-09-29T23:10:17.463 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a0b499700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a0409b770 0x7f3a040630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:17.463 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a0b499700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3a040b3cf0 0x7f3a0414b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:17.463 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a0b499700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a04141810 con 0x7f3a04141440 2023-09-29T23:10:17.463 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a0b499700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0409a7f0 con 0x7f3a040b3cf0 2023-09-29T23:10:17.463 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a0b499700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040a4df0 con 0x7f3a0409b770 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a09235700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f3a04141440 0x7f3a040a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54942/0 (socket says 172.21.15.121:54942) 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a09235700 1 -- 172.21.15.121:0/271262143 learned_addr learned my addr 172.21.15.121:0/271262143 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a08a34700 1 --2- 172.21.15.121:0/271262143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a0409b770 0x7f3a040630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a09a36700 1 --2- 172.21.15.121:0/271262143 >> v2:172.21.15.123:3300/0 conn(0x7f3a040b3cf0 0x7f3a0414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f39fbfff700 1 -- 172.21.15.121:0/271262143 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935644591 0 0) 0x7f3a04141810 con 0x7f3a04141440 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f39fbfff700 1 -- 172.21.15.121:0/271262143 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f0003570 con 0x7f3a04141440 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a08a34700 1 -- 172.21.15.121:0/271262143 >> v2:172.21.15.123:3300/0 conn(0x7f3a040b3cf0 msgr2=0x7f3a0414b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a08a34700 1 --2- 172.21.15.121:0/271262143 >> v2:172.21.15.123:3300/0 conn(0x7f3a040b3cf0 0x7f3a0414b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a08a34700 1 -- 172.21.15.121:0/271262143 >> v1:172.21.15.121:6789/0 conn(0x7f3a04141440 legacy=0x7f3a040a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a08a34700 1 -- 172.21.15.121:0/271262143 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a040a50b0 con 0x7f3a0409b770 2023-09-29T23:10:17.464 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f3a08a34700 1 --2- 172.21.15.121:0/271262143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a0409b770 0x7f3a040630e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39f40098e0 tx=0x7f39f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eee1fae2b0c89db server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f39fbfff700 1 -- 172.21.15.121:0/271262143 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f400b640 con 0x7f3a0409b770 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f39fbfff700 1 -- 172.21.15.121:0/271262143 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f400b7a0 con 0x7f3a0409b770 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.462+0000 7f39fbfff700 1 -- 172.21.15.121:0/271262143 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f400bb40 con 0x7f3a0409b770 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 -- 172.21.15.121:0/271262143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a0409b770 msgr2=0x7f3a040630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 --2- 172.21.15.121:0/271262143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a0409b770 0x7f3a040630e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39f40098e0 tx=0x7f39f40091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 -- 172.21.15.121:0/271262143 shutdown_connections 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 --2- 172.21.15.121:0/271262143 >> v2:172.21.15.123:3300/0 conn(0x7f3a040b3cf0 0x7f3a0414b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 --2- 172.21.15.121:0/271262143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a0409b770 0x7f3a040630e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 -- 172.21.15.121:0/271262143 >> 172.21.15.121:0/271262143 conn(0x7f3a04069e90 msgr2=0x7f3a040b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 -- 172.21.15.121:0/271262143 shutdown_connections 2023-09-29T23:10:17.465 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.463+0000 7f3a0b499700 1 -- 172.21.15.121:0/271262143 wait complete. 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.464+0000 7f3a0b499700 1 Processor -- start 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.464+0000 7f3a0b499700 1 -- start start 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a0b499700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0409b770 0x7f3a040a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a0b499700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a040b3cf0 0x7f3a04150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a0b499700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a04141440 0x7f3a041510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a0b499700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040a15f0 con 0x7f3a040b3cf0 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a0b499700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04153c60 con 0x7f3a0409b770 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a0b499700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04153f00 con 0x7f3a04141440 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a09235700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0409b770 0x7f3a040a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:17.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a09235700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0409b770 0x7f3a040a1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.121:38508/0 (socket says 172.21.15.121:38508) 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a09235700 1 -- 172.21.15.121:0/3016735554 learned_addr learned my addr 172.21.15.121:0/3016735554 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a08a34700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a040b3cf0 0x7f3a04150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a09a36700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a04141440 0x7f3a041510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a08a34700 1 -- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a04141440 msgr2=0x7f3a041510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a08a34700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a04141440 0x7f3a041510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a08a34700 1 -- 172.21.15.121:0/3016735554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0409b770 msgr2=0x7f3a040a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a08a34700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0409b770 0x7f3a040a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a08a34700 1 -- 172.21.15.121:0/3016735554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4009420 con 0x7f3a040b3cf0 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a09235700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0409b770 0x7f3a040a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.465+0000 7f3a08a34700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a040b3cf0 0x7f3a04150b60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f39f4016000 tx=0x7f39f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:17.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.466+0000 7f39f97fa700 1 -- 172.21.15.121:0/3016735554 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f401f070 con 0x7f3a040b3cf0 2023-09-29T23:10:17.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.466+0000 7f39f97fa700 1 -- 172.21.15.121:0/3016735554 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f401a910 con 0x7f3a040b3cf0 2023-09-29T23:10:17.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.466+0000 7f39f97fa700 1 -- 172.21.15.121:0/3016735554 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f401acd0 con 0x7f3a040b3cf0 2023-09-29T23:10:17.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.466+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a04144180 con 0x7f3a040b3cf0 2023-09-29T23:10:17.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.466+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a040a2b20 con 0x7f3a040b3cf0 2023-09-29T23:10:17.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.467+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0406ab60 con 0x7f3a040b3cf0 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.468+0000 7f39f97fa700 1 -- 172.21.15.121:0/3016735554 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f39f4023070 con 0x7f3a040b3cf0 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.468+0000 7f39f97fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.468+0000 7f39f97fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.468+0000 7f39f97fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.468+0000 7f39f97fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.468+0000 7f39f97fa700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f39e80987b0 0x7f39e809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.469+0000 7f3a09235700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f39e80987b0 0x7f39e809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.469+0000 7f39f97fa700 1 -- 172.21.15.121:0/3016735554 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f39f40c19f0 con 0x7f3a040b3cf0 2023-09-29T23:10:17.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.469+0000 7f3a09235700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f39e80987b0 0x7f39e809ac60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a000010e0 tx=0x7f3a00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:17.473 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.472+0000 7f39f97fa700 1 -- 172.21.15.121:0/3016735554 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39f408f0a0 con 0x7f3a040b3cf0 2023-09-29T23:10:17.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.698+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3a04144410 con 0x7f3a040b3cf0 2023-09-29T23:10:17.698 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.698+0000 7f39f97fa700 1 -- 172.21.15.121:0/3016735554 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f39f40926c0 con 0x7f3a040b3cf0 2023-09-29T23:10:17.698 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:17.698 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:17.700 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 10 mgrc shutdown 2023-09-29T23:10:17.700 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f39e80987b0 msgr2=0x7f39e809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:17.700 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f39e80987b0 0x7f39e809ac60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a000010e0 tx=0x7f3a00009040 comp rx=0 tx=0).stop 2023-09-29T23:10:17.700 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a040b3cf0 msgr2=0x7f3a04150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a040b3cf0 0x7f3a04150b60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f39f4016000 tx=0x7f39f4017000 comp rx=0 tx=0).stop 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 shutdown_connections 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f39e80987b0 0x7f39e809ac60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a04141440 0x7f3a041510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a0409b770 0x7f3a040a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 --2- 172.21.15.121:0/3016735554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a040b3cf0 0x7f3a04150b60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 >> 172.21.15.121:0/3016735554 conn(0x7f3a04069e90 msgr2=0x7f3a040a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 shutdown_connections 2023-09-29T23:10:17.701 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:17.700+0000 7f3a0b499700 1 -- 172.21.15.121:0/3016735554 wait complete. 2023-09-29T23:10:20.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.841+0000 7fab99253700 1 Processor -- start 2023-09-29T23:10:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.841+0000 7fab99253700 1 -- start start 2023-09-29T23:10:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab99253700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fab94141440 0x7fab940a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab99253700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9409b770 0x7fab940630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab99253700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab94141810 con 0x7fab940b3cf0 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab99253700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9409a7f0 con 0x7fab94141440 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab99253700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab940a4df0 con 0x7fab9409b770 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab9359e700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fab940b3cf0 0x7fab94143df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54958/0 (socket says 172.21.15.121:54958) 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab9359e700 1 -- 172.21.15.121:0/4038864534 learned_addr learned my addr 172.21.15.121:0/4038864534 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab9259c700 1 --2- 172.21.15.121:0/4038864534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9409b770 0x7fab940630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab92d9d700 1 --2- 172.21.15.121:0/4038864534 >> v2:172.21.15.123:3300/0 conn(0x7fab94141440 0x7fab940a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab91d9b700 1 -- 172.21.15.121:0/4038864534 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969098486 0 0) 0x7fab94141810 con 0x7fab940b3cf0 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab91d9b700 1 -- 172.21.15.121:0/4038864534 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab80003570 con 0x7fab940b3cf0 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab9259c700 1 -- 172.21.15.121:0/4038864534 >> v2:172.21.15.123:3300/0 conn(0x7fab94141440 msgr2=0x7fab940a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab9259c700 1 --2- 172.21.15.121:0/4038864534 >> v2:172.21.15.123:3300/0 conn(0x7fab94141440 0x7fab940a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab9259c700 1 -- 172.21.15.121:0/4038864534 >> v1:172.21.15.121:6789/0 conn(0x7fab940b3cf0 legacy=0x7fab94143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.842+0000 7fab9259c700 1 -- 172.21.15.121:0/4038864534 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab940a50b0 con 0x7fab9409b770 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab9259c700 1 --2- 172.21.15.121:0/4038864534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9409b770 0x7fab940630e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab840098e0 tx=0x7fab840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30512883ff31f704 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab91d9b700 1 -- 172.21.15.121:0/4038864534 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84011630 con 0x7fab9409b770 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab91d9b700 1 -- 172.21.15.121:0/4038864534 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab84011790 con 0x7fab9409b770 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab91d9b700 1 -- 172.21.15.121:0/4038864534 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84011b30 con 0x7fab9409b770 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 -- 172.21.15.121:0/4038864534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9409b770 msgr2=0x7fab940630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 --2- 172.21.15.121:0/4038864534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9409b770 0x7fab940630e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab840098e0 tx=0x7fab840091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 -- 172.21.15.121:0/4038864534 shutdown_connections 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 --2- 172.21.15.121:0/4038864534 >> v2:172.21.15.123:3300/0 conn(0x7fab94141440 0x7fab940a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 --2- 172.21.15.121:0/4038864534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab9409b770 0x7fab940630e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 -- 172.21.15.121:0/4038864534 >> 172.21.15.121:0/4038864534 conn(0x7fab94069e90 msgr2=0x7fab940b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 -- 172.21.15.121:0/4038864534 shutdown_connections 2023-09-29T23:10:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.843+0000 7fab99253700 1 -- 172.21.15.121:0/4038864534 wait complete. 2023-09-29T23:10:20.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.844+0000 7fab99253700 1 Processor -- start 2023-09-29T23:10:20.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.844+0000 7fab99253700 1 -- start start 2023-09-29T23:10:20.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab99253700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9409b770 0x7fab9409fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:20.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab99253700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab940b3cf0 0x7fab9409ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:20.848 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab99253700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab94141440 0x7fab9406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab99253700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9409e1d0 con 0x7fab94141440 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab99253700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab940a25b0 con 0x7fab9409b770 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab99253700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab940a0420 con 0x7fab940b3cf0 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab940b3cf0 0x7fab9409ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab94141440 0x7fab9406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab940b3cf0 0x7fab9409ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:49846/0 (socket says 172.21.15.121:49846) 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 -- 172.21.15.121:0/961855493 learned_addr learned my addr 172.21.15.121:0/961855493 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab92d9d700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9409b770 0x7fab9409fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:20.849 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 -- 172.21.15.121:0/961855493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9409b770 msgr2=0x7fab9409fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9409b770 0x7fab9409fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 -- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab94141440 msgr2=0x7fab9406e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab94141440 0x7fab9406e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.845+0000 7fab9259c700 1 -- 172.21.15.121:0/961855493 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84009420 con 0x7fab940b3cf0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.846+0000 7fab9259c700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab940b3cf0 0x7fab9409ec20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab940a4f30 tx=0x7fab84017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.846+0000 7fab8affd700 1 -- 172.21.15.121:0/961855493 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84000d10 con 0x7fab940b3cf0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.846+0000 7fab8affd700 1 -- 172.21.15.121:0/961855493 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab84011b10 con 0x7fab940b3cf0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.846+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab940a06c0 con 0x7fab940b3cf0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.846+0000 7fab8affd700 1 -- 172.21.15.121:0/961855493 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84025680 con 0x7fab940b3cf0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.846+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab940a18e0 con 0x7fab940b3cf0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.847+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9406a260 con 0x7fab940b3cf0 2023-09-29T23:10:20.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.848+0000 7fab8affd700 1 -- 172.21.15.121:0/961855493 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fab84025a30 con 0x7fab940b3cf0 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.848+0000 7fab8affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.848+0000 7fab8affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.848+0000 7fab8affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.848+0000 7fab8affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.848+0000 7fab8affd700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fab74098780 0x7fab7409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.849+0000 7fab92d9d700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fab74098780 0x7fab7409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.849+0000 7fab8affd700 1 -- 172.21.15.121:0/961855493 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fab840c1ca0 con 0x7fab940b3cf0 2023-09-29T23:10:20.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.849+0000 7fab92d9d700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fab74098780 0x7fab7409ac30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009390 tx=0x7fab7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:20.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:20.853+0000 7fab8affd700 1 -- 172.21.15.121:0/961855493 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab8408f350 con 0x7fab940b3cf0 2023-09-29T23:10:21.074 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.074+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fab9406ab60 con 0x7fab940b3cf0 2023-09-29T23:10:21.074 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.074+0000 7fab8affd700 1 -- 172.21.15.121:0/961855493 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fab84092970 con 0x7fab940b3cf0 2023-09-29T23:10:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:21.075 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:21.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 10 mgrc shutdown 2023-09-29T23:10:21.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fab74098780 msgr2=0x7fab7409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:21.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fab74098780 0x7fab7409ac30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009390 tx=0x7fab7c008040 comp rx=0 tx=0).stop 2023-09-29T23:10:21.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab940b3cf0 msgr2=0x7fab9409ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:21.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab940b3cf0 0x7fab9409ec20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab940a4f30 tx=0x7fab84017000 comp rx=0 tx=0).stop 2023-09-29T23:10:21.077 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 shutdown_connections 2023-09-29T23:10:21.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fab74098780 0x7fab7409ac30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:21.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab94141440 0x7fab9406e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:21.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9409b770 0x7fab9409fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:21.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 --2- 172.21.15.121:0/961855493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab940b3cf0 0x7fab9409ec20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:21.078 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 >> 172.21.15.121:0/961855493 conn(0x7fab94069e90 msgr2=0x7fab940a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:21.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 shutdown_connections 2023-09-29T23:10:21.079 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:21.076+0000 7fab99253700 1 -- 172.21.15.121:0/961855493 wait complete. 2023-09-29T23:10:24.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.220+0000 7f5ee1153700 1 Processor -- start 2023-09-29T23:10:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.220+0000 7f5ee1153700 1 -- start start 2023-09-29T23:10:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5ee1153700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a6950 0x7f5edc0a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5ee1153700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5edc1458e0 0x7f5edc147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5ee1153700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc0a85e0 con 0x7f5edc0a73b0 2023-09-29T23:10:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5ee1153700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5edc0a6d20 con 0x7f5edc1458e0 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5ee1153700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc0a91a0 con 0x7f5edc0a6950 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a6950 0x7f5edc0a8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a6950 0x7f5edc0a8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:49848/0 (socket says 172.21.15.121:49848) 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 -- 172.21.15.121:0/1024475434 learned_addr learned my addr 172.21.15.121:0/1024475434 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edb59e700 1 --2- 172.21.15.121:0/1024475434 >> v2:172.21.15.123:3300/0 conn(0x7f5edc1458e0 0x7f5edc147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 -- 172.21.15.121:0/1024475434 >> v2:172.21.15.123:3300/0 conn(0x7f5edc1458e0 msgr2=0x7f5edc147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 --2- 172.21.15.121:0/1024475434 >> v2:172.21.15.123:3300/0 conn(0x7f5edc1458e0 0x7f5edc147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 -- 172.21.15.121:0/1024475434 >> v1:172.21.15.123:6789/0 conn(0x7f5edc0a73b0 legacy=0x7f5edc13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 -- 172.21.15.121:0/1024475434 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc0a9440 con 0x7f5edc0a6950 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5edad9d700 1 --2- 172.21.15.121:0/1024475434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a6950 0x7f5edc0a8ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000bc50 tx=0x7f5ed000b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7bda85d89fd3cc7 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5ed9d9b700 1 -- 172.21.15.121:0/1024475434 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed000de90 con 0x7f5edc0a6950 2023-09-29T23:10:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.221+0000 7f5ed9d9b700 1 -- 172.21.15.121:0/1024475434 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed000be60 con 0x7f5edc0a6950 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ed9d9b700 1 -- 172.21.15.121:0/1024475434 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00196b0 con 0x7f5edc0a6950 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 -- 172.21.15.121:0/1024475434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a6950 msgr2=0x7f5edc0a8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 --2- 172.21.15.121:0/1024475434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a6950 0x7f5edc0a8ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000bc50 tx=0x7f5ed000b4f0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 -- 172.21.15.121:0/1024475434 shutdown_connections 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 --2- 172.21.15.121:0/1024475434 >> v2:172.21.15.123:3300/0 conn(0x7f5edc1458e0 0x7f5edc147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 --2- 172.21.15.121:0/1024475434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a6950 0x7f5edc0a8ae0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 -- 172.21.15.121:0/1024475434 >> 172.21.15.121:0/1024475434 conn(0x7f5edc06c010 msgr2=0x7f5edc06ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 -- 172.21.15.121:0/1024475434 shutdown_connections 2023-09-29T23:10:24.224 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.222+0000 7f5ee1153700 1 -- 172.21.15.121:0/1024475434 wait complete. 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.223+0000 7f5ee1153700 1 Processor -- start 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.223+0000 7f5ee1153700 1 -- start start 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5ee1153700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5edc0a6950 0x7f5edc0b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5ee1153700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a73b0 0x7f5edc06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5ee1153700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc1458e0 0x7f5edc0aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5ee1153700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc153e70 con 0x7f5edc0a6950 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5ee1153700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc154130 con 0x7f5edc1458e0 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5ee1153700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc1543f0 con 0x7f5edc0a73b0 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5edad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5edc0a6950 0x7f5edc0b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5eda59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a73b0 0x7f5edc06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5edad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5edc0a6950 0x7f5edc0b0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52624/0 (socket says 172.21.15.121:52624) 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5edad9d700 1 -- 172.21.15.121:0/3410243979 learned_addr learned my addr 172.21.15.121:0/3410243979 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5edb59e700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc1458e0 0x7f5edc0aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5eda59c700 1 -- 172.21.15.121:0/3410243979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc1458e0 msgr2=0x7f5edc0aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5eda59c700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc1458e0 0x7f5edc0aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5eda59c700 1 -- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5edc0a6950 msgr2=0x7f5edc0b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5eda59c700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5edc0a6950 0x7f5edc0b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5eda59c700 1 -- 172.21.15.121:0/3410243979 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed000b790 con 0x7f5edc0a73b0 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.224+0000 7f5edad9d700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5edc0a6950 0x7f5edc0b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:24.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.225+0000 7f5eda59c700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a73b0 0x7f5edc06e840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00c590 tx=0x7f5ecc007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.225+0000 7f5ec2ffd700 1 -- 172.21.15.121:0/3410243979 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc011590 con 0x7f5edc0a73b0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.225+0000 7f5ec2ffd700 1 -- 172.21.15.121:0/3410243979 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc011710 con 0x7f5edc0a73b0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.225+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5edc06ed80 con 0x7f5edc0a73b0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.225+0000 7f5ec2ffd700 1 -- 172.21.15.121:0/3410243979 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc011ad0 con 0x7f5edc0a73b0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.225+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5edc1539e0 con 0x7f5edc0a73b0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.226+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc071660 con 0x7f5edc0a73b0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5ec2ffd700 1 -- 172.21.15.121:0/3410243979 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5ecc01b3b0 con 0x7f5edc0a73b0 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5ec2ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5ec2ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5ec2ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5ec2ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5ec2ffd700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5ec8098780 0x7f5ec809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5ec2ffd700 1 -- 172.21.15.121:0/3410243979 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f5ecc0bf830 con 0x7f5edc0a73b0 2023-09-29T23:10:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.227+0000 7f5edad9d700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5ec8098780 0x7f5ec809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:24.230 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.228+0000 7f5edad9d700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5ec8098780 0x7f5ec809ac30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0006e40 tx=0x7f5ed001b000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:24.231 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.231+0000 7f5ec2ffd700 1 -- 172.21.15.121:0/3410243979 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ecc08cee0 con 0x7f5edc0a73b0 2023-09-29T23:10:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.453+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5edc068490 con 0x7f5edc0a73b0 2023-09-29T23:10:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.453+0000 7f5ec2ffd700 1 -- 172.21.15.121:0/3410243979 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5ecc017020 con 0x7f5edc0a73b0 2023-09-29T23:10:24.453 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:24.453 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 10 mgrc shutdown 2023-09-29T23:10:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5ec8098780 msgr2=0x7f5ec809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5ec8098780 0x7f5ec809ac30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0006e40 tx=0x7f5ed001b000 comp rx=0 tx=0).stop 2023-09-29T23:10:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a73b0 msgr2=0x7f5edc06e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a73b0 0x7f5edc06e840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00c590 tx=0x7f5ecc007ec0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 shutdown_connections 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f5ec8098780 0x7f5ec809ac30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5edc1458e0 0x7f5edc0aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5edc0a6950 0x7f5edc0b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 --2- 172.21.15.121:0/3410243979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5edc0a73b0 0x7f5edc06e840 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 >> 172.21.15.121:0/3410243979 conn(0x7f5edc06c010 msgr2=0x7f5edc0a7840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 shutdown_connections 2023-09-29T23:10:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:24.455+0000 7f5ee1153700 1 -- 172.21.15.121:0/3410243979 wait complete. 2023-09-29T23:10:27.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:27.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.598+0000 7f2bcc295700 1 Processor -- start 2023-09-29T23:10:27.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.598+0000 7f2bcc295700 1 -- start start 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bcc295700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2bc40634b0 0x7f2bc41458f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bcc295700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc409e600 con 0x7f2bc409f580 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bcc295700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc40a2d20 con 0x7f2bc40630e0 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bcc295700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc409d8d0 con 0x7f2bc40634b0 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc9830700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2bc40634b0 0x7f2bc41458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc9830700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2bc40634b0 0x7f2bc41458f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53672/0 (socket says 172.21.15.121:53672) 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc9830700 1 -- 172.21.15.121:0/3183659386 learned_addr learned my addr 172.21.15.121:0/3183659386 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bca832700 1 --1- 172.21.15.121:0/3183659386 >> v1:172.21.15.121:6789/0 conn(0x7f2bc409f580 0x7f2bc4147e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:40732/0 (socket says 172.21.15.121:40732) 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc9830700 1 -- 172.21.15.121:0/3183659386 >> v1:172.21.15.123:6789/0 conn(0x7f2bc40630e0 legacy=0x7f2bc409c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:27.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc9830700 1 -- 172.21.15.121:0/3183659386 >> v1:172.21.15.121:6789/0 conn(0x7f2bc409f580 legacy=0x7f2bc4147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc9830700 1 -- 172.21.15.121:0/3183659386 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc409db70 con 0x7f2bc40634b0 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc9830700 1 --2- 172.21.15.121:0/3183659386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc40634b0 0x7f2bc41458f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40099b0 tx=0x7f2bb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfaf348630892641 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc902f700 1 -- 172.21.15.121:0/3183659386 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb400b860 con 0x7f2bc40634b0 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc902f700 1 -- 172.21.15.121:0/3183659386 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb400b9c0 con 0x7f2bc40634b0 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.599+0000 7f2bc902f700 1 -- 172.21.15.121:0/3183659386 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb400bd60 con 0x7f2bc40634b0 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.600+0000 7f2bcc295700 1 -- 172.21.15.121:0/3183659386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc40634b0 msgr2=0x7f2bc41458f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.600+0000 7f2bcc295700 1 --2- 172.21.15.121:0/3183659386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc40634b0 0x7f2bc41458f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40099b0 tx=0x7f2bb4009250 comp rx=0 tx=0).stop 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.600+0000 7f2bcc295700 1 -- 172.21.15.121:0/3183659386 shutdown_connections 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.600+0000 7f2bcc295700 1 --2- 172.21.15.121:0/3183659386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc40634b0 0x7f2bc41458f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.600+0000 7f2bcc295700 1 -- 172.21.15.121:0/3183659386 >> 172.21.15.121:0/3183659386 conn(0x7f2bc4069e90 msgr2=0x7f2bc40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:27.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.600+0000 7f2bcc295700 1 -- 172.21.15.121:0/3183659386 shutdown_connections 2023-09-29T23:10:27.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.600+0000 7f2bcc295700 1 -- 172.21.15.121:0/3183659386 wait complete. 2023-09-29T23:10:27.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 Processor -- start 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 -- start start 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40630e0 0x7f2bc413dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2bc40634b0 0x7f2bc406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc409f580 0x7f2bc406e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc406e850 con 0x7f2bc409f580 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc406ea60 con 0x7f2bc40630e0 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.601+0000 7f2bcc295700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc406ed20 con 0x7f2bc40634b0 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2bc40634b0 0x7f2bc406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bca832700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc409f580 0x7f2bc406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2bc40634b0 0x7f2bc406ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36256/0 (socket says 172.21.15.121:36256) 2023-09-29T23:10:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 -- 172.21.15.121:0/4185519352 learned_addr learned my addr 172.21.15.121:0/4185519352 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bca031700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40630e0 0x7f2bc413dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 -- 172.21.15.121:0/4185519352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40630e0 msgr2=0x7f2bc413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40630e0 0x7f2bc413dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 -- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc409f580 msgr2=0x7f2bc406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc409f580 0x7f2bc406e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 -- 172.21.15.121:0/4185519352 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb40094f0 con 0x7f2bc40634b0 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bca832700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc409f580 0x7f2bc406e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bc9830700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2bc40634b0 0x7f2bc406ddd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2bc406ad60 tx=0x7f2bb4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bba7fc700 1 -- 172.21.15.121:0/4185519352 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4024440 con 0x7f2bc40634b0 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bba7fc700 1 -- 172.21.15.121:0/4185519352 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb40245a0 con 0x7f2bc40634b0 2023-09-29T23:10:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bba7fc700 1 -- 172.21.15.121:0/4185519352 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4024920 con 0x7f2bc40634b0 2023-09-29T23:10:27.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc413e3e0 con 0x7f2bc40634b0 2023-09-29T23:10:27.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.602+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc40a1aa0 con 0x7f2bc40634b0 2023-09-29T23:10:27.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.603+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4144a90 con 0x7f2bc40634b0 2023-09-29T23:10:27.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.604+0000 7f2bba7fc700 1 -- 172.21.15.121:0/4185519352 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2bb401a420 con 0x7f2bc40634b0 2023-09-29T23:10:27.607 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.604+0000 7f2bba7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.605+0000 7f2bba7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.605+0000 7f2bba7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.605+0000 7f2bba7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.605+0000 7f2bba7fc700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ba8098800 0x7f2ba809acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.605+0000 7f2bca031700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ba8098800 0x7f2ba809acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.605+0000 7f2bba7fc700 1 -- 172.21.15.121:0/4185519352 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f2bb40c22d0 con 0x7f2bc40634b0 2023-09-29T23:10:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.605+0000 7f2bca031700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ba8098800 0x7f2ba809acb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0007fd0 tx=0x7f2bc0007eb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.609+0000 7f2bba7fc700 1 -- 172.21.15.121:0/4185519352 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2bc4144a90 con 0x7f2bc40634b0 2023-09-29T23:10:27.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.832+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2bc4144a90 con 0x7f2bc40634b0 2023-09-29T23:10:27.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.832+0000 7f2bba7fc700 1 -- 172.21.15.121:0/4185519352 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2bc4144a90 con 0x7f2bc40634b0 2023-09-29T23:10:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:27.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 10 mgrc shutdown 2023-09-29T23:10:27.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ba8098800 msgr2=0x7f2ba809acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ba8098800 0x7f2ba809acb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0007fd0 tx=0x7f2bc0007eb0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2bc40634b0 msgr2=0x7f2bc406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2bc40634b0 0x7f2bc406ddd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2bc406ad60 tx=0x7f2bb4017000 comp rx=0 tx=0).stop 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 shutdown_connections 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2ba8098800 0x7f2ba809acb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bc409f580 0x7f2bc406e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40630e0 0x7f2bc413dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 --2- 172.21.15.121:0/4185519352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2bc40634b0 0x7f2bc406ddd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 >> 172.21.15.121:0/4185519352 conn(0x7f2bc4069e90 msgr2=0x7f2bc409ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:27.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 shutdown_connections 2023-09-29T23:10:27.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:27.834+0000 7f2bcc295700 1 -- 172.21.15.121:0/4185519352 wait complete. 2023-09-29T23:10:30.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.980+0000 7fdd92bc1700 1 Processor -- start 2023-09-29T23:10:30.981 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.980+0000 7fdd92bc1700 1 -- start start 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd92bc1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd8c09b770 0x7fdd8c0630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd92bc1700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdd8c0b3cf0 0x7fdd8c14b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd92bc1700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8c141810 con 0x7fdd8c141440 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd92bc1700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8c09a7f0 con 0x7fdd8c09b770 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd92bc1700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8c0a4df0 con 0x7fdd8c0b3cf0 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd8c09b770 0x7fdd8c0630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd9095d700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdd8c141440 0x7fdd8c0a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:40748/0 (socket says 172.21.15.121:40748) 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd9095d700 1 -- 172.21.15.121:0/2201238946 learned_addr learned my addr 172.21.15.121:0/2201238946 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd9115e700 1 --2- 172.21.15.121:0/2201238946 >> v2:172.21.15.123:3300/0 conn(0x7fdd8c0b3cf0 0x7fdd8c14b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8bfff700 1 -- 172.21.15.121:0/2201238946 >> v2:172.21.15.123:3300/0 conn(0x7fdd8c0b3cf0 msgr2=0x7fdd8c14b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:30.984 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8bfff700 1 --2- 172.21.15.121:0/2201238946 >> v2:172.21.15.123:3300/0 conn(0x7fdd8c0b3cf0 0x7fdd8c14b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8bfff700 1 -- 172.21.15.121:0/2201238946 >> v1:172.21.15.121:6789/0 conn(0x7fdd8c141440 legacy=0x7fdd8c0a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8b7fe700 1 -- 172.21.15.121:0/2201238946 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753747705 0 0) 0x7fdd8c141810 con 0x7fdd8c141440 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8bfff700 1 -- 172.21.15.121:0/2201238946 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c0a50b0 con 0x7fdd8c09b770 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8bfff700 1 --2- 172.21.15.121:0/2201238946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c09b770 0x7fdd8c0630e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c06b690 tx=0x7fdd780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae12bfa6e9809b05 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8b7fe700 1 -- 172.21.15.121:0/2201238946 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7800b6b0 con 0x7fdd8c09b770 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8b7fe700 1 -- 172.21.15.121:0/2201238946 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7800b810 con 0x7fdd8c09b770 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.981+0000 7fdd8b7fe700 1 -- 172.21.15.121:0/2201238946 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7800bbb0 con 0x7fdd8c09b770 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 -- 172.21.15.121:0/2201238946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c09b770 msgr2=0x7fdd8c0630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/2201238946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c09b770 0x7fdd8c0630e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c06b690 tx=0x7fdd780091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 -- 172.21.15.121:0/2201238946 shutdown_connections 2023-09-29T23:10:30.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/2201238946 >> v2:172.21.15.123:3300/0 conn(0x7fdd8c0b3cf0 0x7fdd8c14b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:30.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/2201238946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c09b770 0x7fdd8c0630e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:30.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 -- 172.21.15.121:0/2201238946 >> 172.21.15.121:0/2201238946 conn(0x7fdd8c069e90 msgr2=0x7fdd8c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:30.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 -- 172.21.15.121:0/2201238946 shutdown_connections 2023-09-29T23:10:30.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.982+0000 7fdd92bc1700 1 -- 172.21.15.121:0/2201238946 wait complete. 2023-09-29T23:10:30.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.983+0000 7fdd92bc1700 1 Processor -- start 2023-09-29T23:10:30.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.983+0000 7fdd92bc1700 1 -- start start 2023-09-29T23:10:30.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd8c09b770 0x7fdd8c143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd8c0b3cf0 0x7fdd8c141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c141440 0x7fdd8c142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c142950 con 0x7fdd8c141440 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c0a2f20 con 0x7fdd8c0b3cf0 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c150280 con 0x7fdd8c09b770 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd8c09b770 0x7fdd8c143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9115e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c141440 0x7fdd8c142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd8c09b770 0x7fdd8c143c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36270/0 (socket says 172.21.15.121:36270) 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 -- 172.21.15.121:0/1045724501 learned_addr learned my addr 172.21.15.121:0/1045724501 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:30.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd8bfff700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd8c0b3cf0 0x7fdd8c141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 -- 172.21.15.121:0/1045724501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd8c0b3cf0 msgr2=0x7fdd8c141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd8c0b3cf0 0x7fdd8c141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 -- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c141440 msgr2=0x7fdd8c142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c141440 0x7fdd8c142410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 -- 172.21.15.121:0/1045724501 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78009420 con 0x7fdd8c09b770 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9115e700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c141440 0x7fdd8c142410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd9095d700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd8c09b770 0x7fdd8c143c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000c5c0 tx=0x7fdd80007e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd88ff9700 1 -- 172.21.15.121:0/1045724501 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd800072e0 con 0x7fdd8c09b770 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd88ff9700 1 -- 172.21.15.121:0/1045724501 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd80007480 con 0x7fdd8c09b770 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd88ff9700 1 -- 172.21.15.121:0/1045724501 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd800057f0 con 0x7fdd8c09b770 2023-09-29T23:10:30.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c142b60 con 0x7fdd8c09b770 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.984+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c0a22b0 con 0x7fdd8c09b770 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.985+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c06ab60 con 0x7fdd8c09b770 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.986+0000 7fdd88ff9700 1 -- 172.21.15.121:0/1045724501 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fdd80005ba0 con 0x7fdd8c09b770 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.986+0000 7fdd88ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.987+0000 7fdd88ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.987+0000 7fdd88ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.987+0000 7fdd88ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.987+0000 7fdd88ff9700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd70098780 0x7fdd7009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.987+0000 7fdd8bfff700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd70098780 0x7fdd7009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.987+0000 7fdd88ff9700 1 -- 172.21.15.121:0/1045724501 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fdd800bf800 con 0x7fdd8c09b770 2023-09-29T23:10:30.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.988+0000 7fdd8bfff700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd70098780 0x7fdd7009ac30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c06b690 tx=0x7fdd78017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:30.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:30.991+0000 7fdd88ff9700 1 -- 172.21.15.121:0/1045724501 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd8004b5a0 con 0x7fdd8c09b770 2023-09-29T23:10:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.215+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd8c071660 con 0x7fdd8c09b770 2023-09-29T23:10:31.216 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.215+0000 7fdd88ff9700 1 -- 172.21.15.121:0/1045724501 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdd8008cd80 con 0x7fdd8c09b770 2023-09-29T23:10:31.217 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:31.217 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.217+0000 7fdd92bc1700 10 mgrc shutdown 2023-09-29T23:10:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd70098780 msgr2=0x7fdd7009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd70098780 0x7fdd7009ac30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c06b690 tx=0x7fdd78017000 comp rx=0 tx=0).stop 2023-09-29T23:10:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd8c09b770 msgr2=0x7fdd8c143c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd8c09b770 0x7fdd8c143c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000c5c0 tx=0x7fdd80007e80 comp rx=0 tx=0).stop 2023-09-29T23:10:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 shutdown_connections 2023-09-29T23:10:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fdd70098780 0x7fdd7009ac30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8c141440 0x7fdd8c142410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd8c09b770 0x7fdd8c143c50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 --2- 172.21.15.121:0/1045724501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd8c0b3cf0 0x7fdd8c141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 >> 172.21.15.121:0/1045724501 conn(0x7fdd8c069e90 msgr2=0x7fdd8c0a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 shutdown_connections 2023-09-29T23:10:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:31.218+0000 7fdd92bc1700 1 -- 172.21.15.121:0/1045724501 wait complete. 2023-09-29T23:10:34.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:34.359 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.359+0000 7fa798e8f700 1 Processor -- start 2023-09-29T23:10:34.360 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.359+0000 7fa798e8f700 1 -- start start 2023-09-29T23:10:34.362 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.359+0000 7fa798e8f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa79406b0b0 0x7fa794148e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:34.362 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.359+0000 7fa798e8f700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa79406f930 0x7fa79414b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:34.362 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.359+0000 7fa798e8f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940a56f0 con 0x7fa79406ace0 2023-09-29T23:10:34.362 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.359+0000 7fa798e8f700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7940688c0 con 0x7fa79406b0b0 2023-09-29T23:10:34.362 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.359+0000 7fa798e8f700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7941403a0 con 0x7fa79406f930 2023-09-29T23:10:34.362 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa791d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa79406b0b0 0x7fa794148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa79259c700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa79406ace0 0x7fa7940a4f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:40752/0 (socket says 172.21.15.121:40752) 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa79259c700 1 -- 172.21.15.121:0/4219387009 learned_addr learned my addr 172.21.15.121:0/4219387009 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa792d9d700 1 --2- 172.21.15.121:0/4219387009 >> v2:172.21.15.123:3300/0 conn(0x7fa79406f930 0x7fa79414b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa791d9b700 1 -- 172.21.15.121:0/4219387009 >> v2:172.21.15.123:3300/0 conn(0x7fa79406f930 msgr2=0x7fa79414b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa791d9b700 1 --2- 172.21.15.121:0/4219387009 >> v2:172.21.15.123:3300/0 conn(0x7fa79406f930 0x7fa79414b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa791d9b700 1 -- 172.21.15.121:0/4219387009 >> v1:172.21.15.121:6789/0 conn(0x7fa79406ace0 legacy=0x7fa7940a4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa791d9b700 1 -- 172.21.15.121:0/4219387009 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa794140600 con 0x7fa79406b0b0 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa79159a700 1 -- 172.21.15.121:0/4219387009 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463525501 0 0) 0x7fa7940a56f0 con 0x7fa79406ace0 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa791d9b700 1 --2- 172.21.15.121:0/4219387009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa794148e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa7940690e0 tx=0x7fa7880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539002fdd92a33bc server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa79159a700 1 -- 172.21.15.121:0/4219387009 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78800b6b0 con 0x7fa79406b0b0 2023-09-29T23:10:34.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa79159a700 1 -- 172.21.15.121:0/4219387009 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78800b810 con 0x7fa79406b0b0 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.360+0000 7fa79159a700 1 -- 172.21.15.121:0/4219387009 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78800bbb0 con 0x7fa79406b0b0 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 -- 172.21.15.121:0/4219387009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 msgr2=0x7fa794148e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 --2- 172.21.15.121:0/4219387009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa794148e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa7940690e0 tx=0x7fa7880091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 -- 172.21.15.121:0/4219387009 shutdown_connections 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 --2- 172.21.15.121:0/4219387009 >> v2:172.21.15.123:3300/0 conn(0x7fa79406f930 0x7fa79414b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 --2- 172.21.15.121:0/4219387009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa794148e10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 -- 172.21.15.121:0/4219387009 >> 172.21.15.121:0/4219387009 conn(0x7fa794067440 msgr2=0x7fa794067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 -- 172.21.15.121:0/4219387009 shutdown_connections 2023-09-29T23:10:34.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.361+0000 7fa798e8f700 1 -- 172.21.15.121:0/4219387009 wait complete. 2023-09-29T23:10:34.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 Processor -- start 2023-09-29T23:10:34.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 -- start start 2023-09-29T23:10:34.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa79406ace0 0x7fa79409ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:34.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa79409fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:34.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79406f930 0x7fa7940a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:34.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79409b160 con 0x7fa79406b0b0 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940a2720 con 0x7fa79406f930 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.362+0000 7fa798e8f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7941536f0 con 0x7fa79406ace0 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa79406ace0 0x7fa79409ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa791d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa79409fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa79406ace0 0x7fa79409ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36286/0 (socket says 172.21.15.121:36286) 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa791d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa79409fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53700/0 (socket says 172.21.15.121:53700) 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 -- 172.21.15.121:0/2634225082 learned_addr learned my addr 172.21.15.121:0/2634225082 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa792d9d700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79406f930 0x7fa7940a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 -- 172.21.15.121:0/2634225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79406f930 msgr2=0x7fa7940a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79406f930 0x7fa7940a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 -- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 msgr2=0x7fa79409fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa79409fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 -- 172.21.15.121:0/2634225082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788009420 con 0x7fa79406ace0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa791d9b700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa79409fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa79259c700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa79406ace0 0x7fa79409ac20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa78400c7e0 tx=0x7fa784007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa7827fc700 1 -- 172.21.15.121:0/2634225082 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784005b30 con 0x7fa79406ace0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa7827fc700 1 -- 172.21.15.121:0/2634225082 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa784005c90 con 0x7fa79406ace0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7941539b0 con 0x7fa79406ace0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa7827fc700 1 -- 172.21.15.121:0/2634225082 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784007300 con 0x7fa79406ace0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.363+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7940a09e0 con 0x7fa79406ace0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.364+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa794071660 con 0x7fa79406ace0 2023-09-29T23:10:34.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.365+0000 7fa7827fc700 1 -- 172.21.15.121:0/2634225082 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fa78401b3b0 con 0x7fa79406ace0 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.365+0000 7fa7827fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.366+0000 7fa7827fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.366+0000 7fa7827fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.366+0000 7fa7827fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.366+0000 7fa7827fc700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa774098800 0x7fa77409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.366+0000 7fa791d9b700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa774098800 0x7fa77409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.366+0000 7fa7827fc700 1 -- 172.21.15.121:0/2634225082 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fa7840bfa40 con 0x7fa79406ace0 2023-09-29T23:10:34.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.366+0000 7fa791d9b700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa774098800 0x7fa77409acb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa79406b480 tx=0x7fa788017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:34.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.370+0000 7fa7827fc700 1 -- 172.21.15.121:0/2634225082 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa78408d050 con 0x7fa79406ace0 2023-09-29T23:10:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.589+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa7940a6c60 con 0x7fa79406ace0 2023-09-29T23:10:34.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.589+0000 7fa7827fc700 1 -- 172.21.15.121:0/2634225082 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa784017100 con 0x7fa79406ace0 2023-09-29T23:10:34.590 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:34.590 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 10 mgrc shutdown 2023-09-29T23:10:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa774098800 msgr2=0x7fa77409acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:34.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa774098800 0x7fa77409acb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa79406b480 tx=0x7fa788017000 comp rx=0 tx=0).stop 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa79406ace0 msgr2=0x7fa79409ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa79406ace0 0x7fa79409ac20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa78400c7e0 tx=0x7fa784007df0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 shutdown_connections 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fa774098800 0x7fa77409acb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79406f930 0x7fa7940a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa79406ace0 0x7fa79409ac20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 --2- 172.21.15.121:0/2634225082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79406b0b0 0x7fa79409fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 >> 172.21.15.121:0/2634225082 conn(0x7fa794067440 msgr2=0x7fa7940b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 shutdown_connections 2023-09-29T23:10:34.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:34.591+0000 7fa798e8f700 1 -- 172.21.15.121:0/2634225082 wait complete. 2023-09-29T23:10:37.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:37.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.733+0000 7f5914bc9700 1 Processor -- start 2023-09-29T23:10:37.734 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.733+0000 7f5914bc9700 1 -- start start 2023-09-29T23:10:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f5914bc9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f59100a4cd0 0x7f59100a5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f5914bc9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5910146c50 0x7f5910149030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f5914bc9700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59100a5110 con 0x7f59101346c0 2023-09-29T23:10:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f5914bc9700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591006e3a0 con 0x7f5910146c50 2023-09-29T23:10:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f5914bc9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591006e640 con 0x7f59100a4cd0 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590ed9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5910146c50 0x7f5910149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590e59c700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f59101346c0 0x7f59100a5e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:47182/0 (socket says 172.21.15.121:47182) 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590ed9d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5910146c50 0x7f5910149030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34442/0 (socket says 172.21.15.121:34442) 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590e59c700 1 -- 172.21.15.121:0/1589548263 learned_addr learned my addr 172.21.15.121:0/1589548263 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590dd9b700 1 --2- 172.21.15.121:0/1589548263 >> v2:172.21.15.123:3300/0 conn(0x7f59100a4cd0 0x7f59100a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590ed9d700 1 -- 172.21.15.121:0/1589548263 >> v2:172.21.15.123:3300/0 conn(0x7f59100a4cd0 msgr2=0x7f59100a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590ed9d700 1 --2- 172.21.15.121:0/1589548263 >> v2:172.21.15.123:3300/0 conn(0x7f59100a4cd0 0x7f59100a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590ed9d700 1 -- 172.21.15.121:0/1589548263 >> v1:172.21.15.121:6789/0 conn(0x7f59101346c0 legacy=0x7f59100a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590ed9d700 1 -- 172.21.15.121:0/1589548263 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59100a8520 con 0x7f5910146c50 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590d59a700 1 -- 172.21.15.121:0/1589548263 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177506992 0 0) 0x7f59100a5110 con 0x7f59101346c0 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.734+0000 7f590ed9d700 1 --2- 172.21.15.121:0/1589548263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5910146c50 0x7f5910149030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5908009e60 tx=0x7f5908009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922f636081918a2 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f590d59a700 1 -- 172.21.15.121:0/1589548263 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590800e070 con 0x7f5910146c50 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f590d59a700 1 -- 172.21.15.121:0/1589548263 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590800bb20 con 0x7f5910146c50 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f590d59a700 1 -- 172.21.15.121:0/1589548263 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908012430 con 0x7f5910146c50 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 -- 172.21.15.121:0/1589548263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5910146c50 msgr2=0x7f5910149030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1589548263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5910146c50 0x7f5910149030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5908009e60 tx=0x7f5908009780 comp rx=0 tx=0).stop 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 -- 172.21.15.121:0/1589548263 shutdown_connections 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1589548263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5910146c50 0x7f5910149030 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1589548263 >> v2:172.21.15.123:3300/0 conn(0x7f59100a4cd0 0x7f59100a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 -- 172.21.15.121:0/1589548263 >> 172.21.15.121:0/1589548263 conn(0x7f5910067440 msgr2=0x7f5910067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 -- 172.21.15.121:0/1589548263 shutdown_connections 2023-09-29T23:10:37.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.735+0000 7f5914bc9700 1 -- 172.21.15.121:0/1589548263 wait complete. 2023-09-29T23:10:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.736+0000 7f5914bc9700 1 Processor -- start 2023-09-29T23:10:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.736+0000 7f5914bc9700 1 -- start start 2023-09-29T23:10:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f5914bc9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59100a4cd0 0x7f5910144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f5914bc9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59101346c0 0x7f5910145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f5914bc9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5910146c50 0x7f5910145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f5914bc9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910146210 con 0x7f59101346c0 2023-09-29T23:10:37.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f5914bc9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910143e80 con 0x7f5910146c50 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f5914bc9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59100686d0 con 0x7f59100a4cd0 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590e59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59100a4cd0 0x7f5910144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59101346c0 0x7f5910145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590e59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59100a4cd0 0x7f5910144560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56612/0 (socket says 172.21.15.121:56612) 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590e59c700 1 -- 172.21.15.121:0/1879130828 learned_addr learned my addr 172.21.15.121:0/1879130828 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590ed9d700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5910146c50 0x7f5910145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590dd9b700 1 -- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59100a4cd0 msgr2=0x7f5910144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590dd9b700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59100a4cd0 0x7f5910144560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590dd9b700 1 -- 172.21.15.121:0/1879130828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5910146c50 msgr2=0x7f5910145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590dd9b700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5910146c50 0x7f5910145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590dd9b700 1 -- 172.21.15.121:0/1879130828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5908009a20 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.737+0000 7f590dd9b700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59101346c0 0x7f5910145790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f590000ea50 tx=0x7f59000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.738+0000 7f58fe7fc700 1 -- 172.21.15.121:0/1879130828 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590000c7f0 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.738+0000 7f58fe7fc700 1 -- 172.21.15.121:0/1879130828 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590000c970 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.738+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5910145250 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.738+0000 7f58fe7fc700 1 -- 172.21.15.121:0/1879130828 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590000cd30 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.738+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59101261b0 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.739+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591006f7e0 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.739+0000 7f58fe7fc700 1 -- 172.21.15.121:0/1879130828 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5900013680 con 0x7f59101346c0 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.739+0000 7f58fe7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.740+0000 7f58fe7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.740+0000 7f58fe7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:37.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.740+0000 7f58fe7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:37.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.740+0000 7f58fe7fc700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f58f0098800 0x7f58f009acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:37.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.740+0000 7f590e59c700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f58f0098800 0x7f58f009acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:37.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.740+0000 7f58fe7fc700 1 -- 172.21.15.121:0/1879130828 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f59000c06e0 con 0x7f59101346c0 2023-09-29T23:10:37.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.741+0000 7f590e59c700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f58f0098800 0x7f58f009acb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5904007fd0 tx=0x7f5904007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:37.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.744+0000 7f58fe7fc700 1 -- 172.21.15.121:0/1879130828 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f590008dcf0 con 0x7f59101346c0 2023-09-29T23:10:37.968 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.967+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5910071660 con 0x7f59101346c0 2023-09-29T23:10:37.968 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.967+0000 7f58fe7fc700 1 -- 172.21.15.121:0/1879130828 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5900091310 con 0x7f59101346c0 2023-09-29T23:10:37.968 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:37.968 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 10 mgrc shutdown 2023-09-29T23:10:37.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f58f0098800 msgr2=0x7f58f009acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f58f0098800 0x7f58f009acb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5904007fd0 tx=0x7f5904007df0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59101346c0 msgr2=0x7f5910145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59101346c0 0x7f5910145790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f590000ea50 tx=0x7f59000091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 shutdown_connections 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f58f0098800 0x7f58f009acb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5910146c50 0x7f5910145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59100a4cd0 0x7f5910144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 --2- 172.21.15.121:0/1879130828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59101346c0 0x7f5910145790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 >> 172.21.15.121:0/1879130828 conn(0x7f5910067440 msgr2=0x7f59100b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 shutdown_connections 2023-09-29T23:10:37.971 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:37.969+0000 7f5914bc9700 1 -- 172.21.15.121:0/1879130828 wait complete. 2023-09-29T23:10:40.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:41.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.110+0000 7efc192b4700 1 Processor -- start 2023-09-29T23:10:41.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.110+0000 7efc192b4700 1 -- start start 2023-09-29T23:10:41.113 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc192b4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1406a3e0 0x7efc140a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc192b4700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efc1406a7b0 0x7efc1406e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc192b4700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc14140700 con 0x7efc1406f5b0 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc192b4700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc140a68c0 con 0x7efc1406a7b0 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc192b4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc140a66f0 con 0x7efc1406a3e0 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc12ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1406a3e0 0x7efc140a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc12ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1406a3e0 0x7efc140a61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:56628/0 (socket says 172.21.15.121:56628) 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc12ffd700 1 -- 172.21.15.121:0/4104349716 learned_addr learned my addr 172.21.15.121:0/4104349716 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc137fe700 1 --2- 172.21.15.121:0/4104349716 >> v2:172.21.15.123:3300/0 conn(0x7efc1406a7b0 0x7efc1406e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc12ffd700 1 -- 172.21.15.121:0/4104349716 >> v2:172.21.15.123:3300/0 conn(0x7efc1406a7b0 msgr2=0x7efc1406e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc12ffd700 1 --2- 172.21.15.121:0/4104349716 >> v2:172.21.15.123:3300/0 conn(0x7efc1406a7b0 0x7efc1406e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc12ffd700 1 -- 172.21.15.121:0/4104349716 >> v1:172.21.15.123:6789/0 conn(0x7efc1406f5b0 legacy=0x7efc14140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc12ffd700 1 -- 172.21.15.121:0/4104349716 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc141409a0 con 0x7efc1406a3e0 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.111+0000 7efc137fe700 1 --2- 172.21.15.121:0/4104349716 >> v2:172.21.15.123:3300/0 conn(0x7efc1406a7b0 0x7efc1406e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc12ffd700 1 --2- 172.21.15.121:0/4104349716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1406a3e0 0x7efc140a61b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc080099b0 tx=0x7efc08009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8b01afb60ba4004 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc11ffb700 1 -- 172.21.15.121:0/4104349716 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0800b820 con 0x7efc1406a3e0 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc11ffb700 1 -- 172.21.15.121:0/4104349716 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0800b980 con 0x7efc1406a3e0 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc11ffb700 1 -- 172.21.15.121:0/4104349716 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0800bd20 con 0x7efc1406a3e0 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 -- 172.21.15.121:0/4104349716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1406a3e0 msgr2=0x7efc140a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 --2- 172.21.15.121:0/4104349716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1406a3e0 0x7efc140a61b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc080099b0 tx=0x7efc08009250 comp rx=0 tx=0).stop 2023-09-29T23:10:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 -- 172.21.15.121:0/4104349716 shutdown_connections 2023-09-29T23:10:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 --2- 172.21.15.121:0/4104349716 >> v2:172.21.15.123:3300/0 conn(0x7efc1406a7b0 0x7efc1406e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 --2- 172.21.15.121:0/4104349716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1406a3e0 0x7efc140a61b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 -- 172.21.15.121:0/4104349716 >> 172.21.15.121:0/4104349716 conn(0x7efc14069020 msgr2=0x7efc14069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 -- 172.21.15.121:0/4104349716 shutdown_connections 2023-09-29T23:10:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.112+0000 7efc192b4700 1 -- 172.21.15.121:0/4104349716 wait complete. 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.113+0000 7efc192b4700 1 Processor -- start 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.113+0000 7efc192b4700 1 -- start start 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc192b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1406a3e0 0x7efc14151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc192b4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1406a7b0 0x7efc14067600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc192b4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1409aa30 0x7efc14098c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc192b4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1409aed0 con 0x7efc1406a3e0 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc192b4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1409a5a0 con 0x7efc1406a7b0 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc192b4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc14099f60 con 0x7efc1409aa30 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc12ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1406a3e0 0x7efc14151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc137fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1409aa30 0x7efc14098c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:41.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc12ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1406a3e0 0x7efc14151570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34472/0 (socket says 172.21.15.121:34472) 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc12ffd700 1 -- 172.21.15.121:0/2801614512 learned_addr learned my addr 172.21.15.121:0/2801614512 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc127fc700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1406a7b0 0x7efc14067600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc137fe700 1 -- 172.21.15.121:0/2801614512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1406a7b0 msgr2=0x7efc14067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc137fe700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1406a7b0 0x7efc14067600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc137fe700 1 -- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1406a3e0 msgr2=0x7efc14151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc137fe700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1406a3e0 0x7efc14151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc137fe700 1 -- 172.21.15.121:0/2801614512 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080094f0 con 0x7efc1409aa30 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc127fc700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1406a7b0 0x7efc14067600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.114+0000 7efc12ffd700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1406a3e0 0x7efc14151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.115+0000 7efc137fe700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1409aa30 0x7efc14098c30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc1406fe00 tx=0x7efc0c007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.115+0000 7efbfaffd700 1 -- 172.21.15.121:0/2801614512 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c015440 con 0x7efc1409aa30 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.115+0000 7efbfaffd700 1 -- 172.21.15.121:0/2801614512 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c0155a0 con 0x7efc1409aa30 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.115+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc14151bc0 con 0x7efc1409aa30 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.115+0000 7efbfaffd700 1 -- 172.21.15.121:0/2801614512 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c015960 con 0x7efc1409aa30 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.115+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1409f920 con 0x7efc1409aa30 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.116+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1409a730 con 0x7efc1409aa30 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efbfaffd700 1 -- 172.21.15.121:0/2801614512 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7efc0c00d440 con 0x7efc1409aa30 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efbfaffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efbfaffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efbfaffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efbfaffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efbfaffd700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efc00098780 0x7efc0009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:41.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efc12ffd700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efc00098780 0x7efc0009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:41.122 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.117+0000 7efbfaffd700 1 -- 172.21.15.121:0/2801614512 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7efc0c0c0f20 con 0x7efc1409aa30 2023-09-29T23:10:41.122 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.118+0000 7efc12ffd700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efc00098780 0x7efc0009ac30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc0800c010 tx=0x7efc08016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:41.122 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.121+0000 7efbfaffd700 1 -- 172.21.15.121:0/2801614512 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc0c08e5d0 con 0x7efc1409aa30 2023-09-29T23:10:41.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.341+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc14071660 con 0x7efc1409aa30 2023-09-29T23:10:41.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.342+0000 7efbfaffd700 1 -- 172.21.15.121:0/2801614512 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efc0c091bf0 con 0x7efc1409aa30 2023-09-29T23:10:41.343 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:41.343 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 10 mgrc shutdown 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efc00098780 msgr2=0x7efc0009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efc00098780 0x7efc0009ac30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc0800c010 tx=0x7efc08016000 comp rx=0 tx=0).stop 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1409aa30 msgr2=0x7efc14098c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1409aa30 0x7efc14098c30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc1406fe00 tx=0x7efc0c007840 comp rx=0 tx=0).stop 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 shutdown_connections 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7efc00098780 0x7efc0009ac30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc1409aa30 0x7efc14098c30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1406a3e0 0x7efc14151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 --2- 172.21.15.121:0/2801614512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1406a7b0 0x7efc14067600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 >> 172.21.15.121:0/2801614512 conn(0x7efc14069020 msgr2=0x7efc14140bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 shutdown_connections 2023-09-29T23:10:41.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:41.344+0000 7efc192b4700 1 -- 172.21.15.121:0/2801614512 wait complete. 2023-09-29T23:10:44.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:44.487 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.485+0000 7fb7eac92700 1 Processor -- start 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.485+0000 7fb7eac92700 1 -- start start 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7eac92700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb7e4141440 0x7fb7e40a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7eac92700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7e40b3cf0 0x7fb7e414b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7eac92700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e4141810 con 0x7fb7e409b770 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7eac92700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e409a7f0 con 0x7fb7e40b3cf0 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7eac92700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e40a4df0 con 0x7fb7e4141440 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e922f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7e40b3cf0 0x7fb7e414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e922f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb7e40b3cf0 0x7fb7e414b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34488/0 (socket says 172.21.15.121:34488) 2023-09-29T23:10:44.488 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e922f700 1 -- 172.21.15.121:0/1285063058 learned_addr learned my addr 172.21.15.121:0/1285063058 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e8a2e700 1 --2- 172.21.15.121:0/1285063058 >> v2:172.21.15.123:3300/0 conn(0x7fb7e4141440 0x7fb7e40a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e922f700 1 -- 172.21.15.121:0/1285063058 >> v2:172.21.15.123:3300/0 conn(0x7fb7e4141440 msgr2=0x7fb7e40a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e922f700 1 --2- 172.21.15.121:0/1285063058 >> v2:172.21.15.123:3300/0 conn(0x7fb7e4141440 0x7fb7e40a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e922f700 1 -- 172.21.15.121:0/1285063058 >> v1:172.21.15.123:6789/0 conn(0x7fb7e409b770 legacy=0x7fb7e40630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e922f700 1 -- 172.21.15.121:0/1285063058 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e40a50b0 con 0x7fb7e40b3cf0 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.486+0000 7fb7e8a2e700 1 --2- 172.21.15.121:0/1285063058 >> v2:172.21.15.123:3300/0 conn(0x7fb7e4141440 0x7fb7e40a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7e922f700 1 --2- 172.21.15.121:0/1285063058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e40b3cf0 0x7fb7e414b190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb7e406bea0 tx=0x7fb7dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce008a99531800ab server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7e37fe700 1 -- 172.21.15.121:0/1285063058 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00e070 con 0x7fb7e40b3cf0 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7e37fe700 1 -- 172.21.15.121:0/1285063058 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc00ba10 con 0x7fb7e40b3cf0 2023-09-29T23:10:44.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7e37fe700 1 -- 172.21.15.121:0/1285063058 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0092e0 con 0x7fb7e40b3cf0 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 -- 172.21.15.121:0/1285063058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e40b3cf0 msgr2=0x7fb7e414b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 --2- 172.21.15.121:0/1285063058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e40b3cf0 0x7fb7e414b190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb7e406bea0 tx=0x7fb7dc009840 comp rx=0 tx=0).stop 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 -- 172.21.15.121:0/1285063058 shutdown_connections 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 --2- 172.21.15.121:0/1285063058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e40b3cf0 0x7fb7e414b190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 --2- 172.21.15.121:0/1285063058 >> v2:172.21.15.123:3300/0 conn(0x7fb7e4141440 0x7fb7e40a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 -- 172.21.15.121:0/1285063058 >> 172.21.15.121:0/1285063058 conn(0x7fb7e4069e90 msgr2=0x7fb7e40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 -- 172.21.15.121:0/1285063058 shutdown_connections 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.487+0000 7fb7eac92700 1 -- 172.21.15.121:0/1285063058 wait complete. 2023-09-29T23:10:44.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.488+0000 7fb7eac92700 1 Processor -- start 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7eac92700 1 -- start start 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7eac92700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e409b770 0x7fb7e41433a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7eac92700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7e40b3cf0 0x7fb7e4142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7eac92700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7e4141440 0x7fb7e40a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7eac92700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e4143d70 con 0x7fb7e409b770 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7eac92700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e4144010 con 0x7fb7e40b3cf0 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7eac92700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e41442b0 con 0x7fb7e4141440 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e409b770 0x7fb7e41433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e409b770 0x7fb7e41433a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34490/0 (socket says 172.21.15.121:34490) 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e922f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7e4141440 0x7fb7e40a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 -- 172.21.15.121:0/2631568020 learned_addr learned my addr 172.21.15.121:0/2631568020 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e3fff700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7e40b3cf0 0x7fb7e4142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 -- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7e4141440 msgr2=0x7fb7e40a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7e4141440 0x7fb7e40a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 -- 172.21.15.121:0/2631568020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7e40b3cf0 msgr2=0x7fb7e4142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7e40b3cf0 0x7fb7e4142350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e8a2e700 1 -- 172.21.15.121:0/2631568020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc009ae0 con 0x7fb7e409b770 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.489+0000 7fb7e922f700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7e4141440 0x7fb7e40a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.490+0000 7fb7e8a2e700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e409b770 0x7fb7e41433a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800c880 tx=0x7fb7d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.490+0000 7fb7e0ff9700 1 -- 172.21.15.121:0/2631568020 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8005a00 con 0x7fb7e409b770 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.490+0000 7fb7e0ff9700 1 -- 172.21.15.121:0/2631568020 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d8005b60 con 0x7fb7e409b770 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.490+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e40a06b0 con 0x7fb7e409b770 2023-09-29T23:10:44.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.490+0000 7fb7e0ff9700 1 -- 172.21.15.121:0/2631568020 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8011600 con 0x7fb7e409b770 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.490+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e413d3b0 con 0x7fb7e409b770 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.491+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e406ab60 con 0x7fb7e409b770 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.492+0000 7fb7e0ff9700 1 -- 172.21.15.121:0/2631568020 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb7d80119f0 con 0x7fb7e409b770 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.492+0000 7fb7e0ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.493+0000 7fb7e0ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.493+0000 7fb7e0ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.493+0000 7fb7e0ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.493+0000 7fb7e0ff9700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7c8098780 0x7fb7c809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.493+0000 7fb7e3fff700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7c8098780 0x7fb7c809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:44.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.493+0000 7fb7e0ff9700 1 -- 172.21.15.121:0/2631568020 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fb7d80c0730 con 0x7fb7e409b770 2023-09-29T23:10:44.498 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.493+0000 7fb7e3fff700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7c8098780 0x7fb7c809ac30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4064000 tx=0x7fb7d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:44.498 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.496+0000 7fb7e0ff9700 1 -- 172.21.15.121:0/2631568020 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb7d808dd50 con 0x7fb7e409b770 2023-09-29T23:10:44.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.715+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb7e4071660 con 0x7fb7e409b770 2023-09-29T23:10:44.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.716+0000 7fb7e0ff9700 1 -- 172.21.15.121:0/2631568020 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb7d801e020 con 0x7fb7e409b770 2023-09-29T23:10:44.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:44.717 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.717+0000 7fb7eac92700 10 mgrc shutdown 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.717+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7c8098780 msgr2=0x7fb7c809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.717+0000 7fb7eac92700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7c8098780 0x7fb7c809ac30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4064000 tx=0x7fb7d0009040 comp rx=0 tx=0).stop 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e409b770 msgr2=0x7fb7e41433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e409b770 0x7fb7e41433a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800c880 tx=0x7fb7d8008040 comp rx=0 tx=0).stop 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 shutdown_connections 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb7c8098780 0x7fb7c809ac30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7e4141440 0x7fb7e40a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7e409b770 0x7fb7e41433a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 --2- 172.21.15.121:0/2631568020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7e40b3cf0 0x7fb7e4142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:44.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 >> 172.21.15.121:0/2631568020 conn(0x7fb7e4069e90 msgr2=0x7fb7e413ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:44.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 shutdown_connections 2023-09-29T23:10:44.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:44.718+0000 7fb7eac92700 1 -- 172.21.15.121:0/2631568020 wait complete. 2023-09-29T23:10:47.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:47.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.858+0000 7f9ddb9b9700 1 Processor -- start 2023-09-29T23:10:47.859 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.858+0000 7f9ddb9b9700 1 -- start start 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9ddb9b9700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9dd4147c60 0x7f9dd414a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9ddb9b9700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd40a5110 con 0x7f9dd40a4cd0 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9ddb9b9700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd40a5c80 con 0x7f9dd41346c0 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9ddb9b9700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd406e3a0 con 0x7f9dd4147c60 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9dd8f54700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9dd40a4cd0 0x7f9dd40a5570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35966/0 (socket says 172.21.15.121:35966) 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9dd8f54700 1 -- 172.21.15.121:0/1876451805 learned_addr learned my addr 172.21.15.121:0/1876451805 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040812414 0 0) 0x7f9dd40a5110 con 0x7f9dd40a4cd0 2023-09-29T23:10:47.863 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9dd9f56700 1 --2- 172.21.15.121:0/1876451805 >> v2:172.21.15.123:3300/0 conn(0x7f9dd4147c60 0x7f9dd414a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.859+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc0003570 con 0x7f9dd40a4cd0 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3016653365 0 0) 0x7f9dc0003570 con 0x7f9dd40a4cd0 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 >> v2:172.21.15.123:3300/0 conn(0x7f9dd4147c60 msgr2=0x7f9dd414a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 --2- 172.21.15.121:0/1876451805 >> v2:172.21.15.123:3300/0 conn(0x7f9dd4147c60 0x7f9dd414a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 >> v1:172.21.15.123:6789/0 conn(0x7f9dd41346c0 legacy=0x7f9dd40a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd406e640 con 0x7f9dd40a4cd0 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7f9dc8001470 con 0x7f9dd40a4cd0 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 >> v1:172.21.15.121:6789/0 conn(0x7f9dd40a4cd0 legacy=0x7f9dd40a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dc0003a90 0x7f9dc0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0004810 0x7f9dc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dc0005210 0x7f9dc000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd406e3a0 con 0x7f9dc0005210 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc000c130 con 0x7f9dc0004810 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc000c2c0 con 0x7f9dc0003a90 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dc0005210 0x7f9dc000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9755700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dc0003a90 0x7f9dc0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd8f54700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0004810 0x7f9dc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 -- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dc0003a90 msgr2=0x7f9dc0003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dc0003a90 0x7f9dc0003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 -- 172.21.15.121:0/1876451805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0004810 msgr2=0x7f9dc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0004810 0x7f9dc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:47.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 -- 172.21.15.121:0/1876451805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0003570 con 0x7f9dc0005210 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 -- 172.21.15.121:0/1876451805 reap_dead start 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dd9f56700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dc0005210 0x7f9dc000ba60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e240 tx=0x7f9dcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0825cd63b331a1a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc009ac0 con 0x7f9dc0005210 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc00f040 con 0x7f9dc0005210 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.860+0000 7f9dc7fff700 1 -- 172.21.15.121:0/1876451805 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc00ed40 con 0x7f9dc0005210 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.861+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dc0005210 msgr2=0x7f9dc000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.861+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dc0005210 0x7f9dc000ba60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e240 tx=0x7f9dcc00b040 comp rx=0 tx=0).stop 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.861+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1876451805 shutdown_connections 2023-09-29T23:10:47.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.861+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1876451805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dc0005210 0x7f9dc000ba60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:47.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.861+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1876451805 >> 172.21.15.121:0/1876451805 conn(0x7f9dd4067440 msgr2=0x7f9dd4067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:47.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.861+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1876451805 shutdown_connections 2023-09-29T23:10:47.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.861+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1876451805 wait complete. 2023-09-29T23:10:47.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.862+0000 7f9ddb9b9700 1 Processor -- start 2023-09-29T23:10:47.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.862+0000 7f9ddb9b9700 1 -- start start 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9ddb9b9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0005210 0x7f9dd4146330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9ddb9b9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dd4146870 0x7f9dd412ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9ddb9b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd412b3c0 0x7f9dd4150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9ddb9b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4145cc0 con 0x7f9dd412b3c0 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9ddb9b9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd406a2d0 con 0x7f9dc0005210 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9ddb9b9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4133000 con 0x7f9dd4146870 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd9f56700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd412b3c0 0x7f9dd4150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd9f56700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd412b3c0 0x7f9dd4150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53934/0 (socket says 172.21.15.121:53934) 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd8f54700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dd4146870 0x7f9dd412ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd9f56700 1 -- 172.21.15.121:0/1932797061 learned_addr learned my addr 172.21.15.121:0/1932797061 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd9755700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0005210 0x7f9dd4146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd8f54700 1 -- 172.21.15.121:0/1932797061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0005210 msgr2=0x7f9dd4146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd8f54700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0005210 0x7f9dd4146330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd8f54700 1 -- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd412b3c0 msgr2=0x7f9dd4150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd8f54700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd412b3c0 0x7f9dd4150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd8f54700 1 -- 172.21.15.121:0/1932797061 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd41503c0 con 0x7f9dd4146870 2023-09-29T23:10:47.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd9f56700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd412b3c0 0x7f9dd4150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd9755700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0005210 0x7f9dd4146330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.863+0000 7f9dd8f54700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dd4146870 0x7f9dd412ae80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80073c0 tx=0x7f9dd406b340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.864+0000 7f9dc57fa700 1 -- 172.21.15.121:0/1932797061 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc800f8d0 con 0x7f9dd4146870 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.864+0000 7f9dc57fa700 1 -- 172.21.15.121:0/1932797061 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc800fa30 con 0x7f9dd4146870 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.864+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd4150120 con 0x7f9dd4146870 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.864+0000 7f9dc57fa700 1 -- 172.21.15.121:0/1932797061 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc800fdf0 con 0x7f9dd4146870 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.864+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd41471c0 con 0x7f9dd4146870 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.864+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd4071660 con 0x7f9dd4146870 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.866+0000 7f9dc57fa700 1 -- 172.21.15.121:0/1932797061 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9dc8013600 con 0x7f9dd4146870 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.866+0000 7f9dc57fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:47.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.867+0000 7f9dc57fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:47.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.867+0000 7f9dc57fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:47.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.867+0000 7f9dc57fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:47.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.867+0000 7f9dc57fa700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9db8098780 0x7f9db809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:47.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.867+0000 7f9dd9755700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9db8098780 0x7f9db809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:47.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.867+0000 7f9dc57fa700 1 -- 172.21.15.121:0/1932797061 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f9dc80c0f30 con 0x7f9dd4146870 2023-09-29T23:10:47.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.867+0000 7f9dd9755700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9db8098780 0x7f9db809ac30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0007e40 tx=0x7f9dd0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:47.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:47.870+0000 7f9dc57fa700 1 -- 172.21.15.121:0/1932797061 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9dc808e5e0 con 0x7f9dd4146870 2023-09-29T23:10:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.090+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9dd406a410 con 0x7f9dd4146870 2023-09-29T23:10:48.091 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.090+0000 7f9dc57fa700 1 -- 172.21.15.121:0/1932797061 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9dc8091c00 con 0x7f9dd4146870 2023-09-29T23:10:48.091 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:48.091 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:48.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 10 mgrc shutdown 2023-09-29T23:10:48.093 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9db8098780 msgr2=0x7f9db809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9db8098780 0x7f9db809ac30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0007e40 tx=0x7f9dd0009040 comp rx=0 tx=0).stop 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dd4146870 msgr2=0x7f9dd412ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dd4146870 0x7f9dd412ae80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80073c0 tx=0x7f9dd406b340 comp rx=0 tx=0).stop 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 shutdown_connections 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9db8098780 0x7f9db809ac30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd412b3c0 0x7f9dd4150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0005210 0x7f9dd4146330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 --2- 172.21.15.121:0/1932797061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dd4146870 0x7f9dd412ae80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.092+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 >> 172.21.15.121:0/1932797061 conn(0x7f9dd4067440 msgr2=0x7f9dd40a71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:48.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.093+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 shutdown_connections 2023-09-29T23:10:48.095 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:48.093+0000 7f9ddb9b9700 1 -- 172.21.15.121:0/1932797061 wait complete. 2023-09-29T23:10:51.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:51.234 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.234+0000 7fd2a3dac700 1 Processor -- start 2023-09-29T23:10:51.235 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.234+0000 7fd2a3dac700 1 -- start start 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a3dac700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c0a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a3dac700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd29c1458e0 0x7fd29c147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a3dac700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29c0a85e0 con 0x7fd29c0a73b0 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a3dac700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c0a6d20 con 0x7fd29c1458e0 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a3dac700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0a91a0 con 0x7fd29c0a6950 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a1b48700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c0a8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd29c1458e0 0x7fd29c147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd29c1458e0 0x7fd29c147cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53944/0 (socket says 172.21.15.121:53944) 2023-09-29T23:10:51.237 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a1347700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd29c0a73b0 0x7fd29c13d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35976/0 (socket says 172.21.15.121:35976) 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a1b48700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c0a8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:51230/0 (socket says 172.21.15.121:51230) 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 -- 172.21.15.121:0/189787680 learned_addr learned my addr 172.21.15.121:0/189787680 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 -- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 msgr2=0x7fd29c0a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 --2- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c0a8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 -- 172.21.15.121:0/189787680 >> v1:172.21.15.121:6789/0 conn(0x7fd29c0a73b0 legacy=0x7fd29c13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a0b46700 1 -- 172.21.15.121:0/189787680 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640602974 0 0) 0x7fd29c0a85e0 con 0x7fd29c0a73b0 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 -- 172.21.15.121:0/189787680 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c0a9440 con 0x7fd29c1458e0 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a1b48700 1 --2- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c0a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a2349700 1 --2- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c147cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd294009f20 tx=0x7fd294009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd2c0b05786545a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a0b46700 1 -- 172.21.15.121:0/189787680 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400e070 con 0x7fd29c1458e0 2023-09-29T23:10:51.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a0b46700 1 -- 172.21.15.121:0/189787680 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29400ba60 con 0x7fd29c1458e0 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.235+0000 7fd2a0b46700 1 -- 172.21.15.121:0/189787680 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2940092e0 con 0x7fd29c1458e0 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 -- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 msgr2=0x7fd29c147cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c147cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd294009f20 tx=0x7fd294009840 comp rx=0 tx=0).stop 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 -- 172.21.15.121:0/189787680 shutdown_connections 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c147cc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/189787680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c0a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 -- 172.21.15.121:0/189787680 >> 172.21.15.121:0/189787680 conn(0x7fd29c06c010 msgr2=0x7fd29c06ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 -- 172.21.15.121:0/189787680 shutdown_connections 2023-09-29T23:10:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.236+0000 7fd2a3dac700 1 -- 172.21.15.121:0/189787680 wait complete. 2023-09-29T23:10:51.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.237+0000 7fd2a3dac700 1 Processor -- start 2023-09-29T23:10:51.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.237+0000 7fd2a3dac700 1 -- start start 2023-09-29T23:10:51.242 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a3dac700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a3dac700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd29c0a73b0 0x7fd29c0af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a3dac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c0af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a3dac700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c144540 con 0x7fd29c1458e0 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a3dac700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0aea10 con 0x7fd29c0a73b0 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a3dac700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c145e30 con 0x7fd29c0a6950 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c143bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:51240/0 (socket says 172.21.15.121:51240) 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 -- 172.21.15.121:0/1730465114 learned_addr learned my addr 172.21.15.121:0/1730465114 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a2349700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c0af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1347700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd29c0a73b0 0x7fd29c0af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:51.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 -- 172.21.15.121:0/1730465114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd29c0a73b0 msgr2=0x7fd29c0af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd29c0a73b0 0x7fd29c0af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 -- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 msgr2=0x7fd29c0af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c0af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 -- 172.21.15.121:0/1730465114 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298009420 con 0x7fd29c0a6950 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1347700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd29c0a73b0 0x7fd29c0af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a2349700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c0af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a1b48700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c143bd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd29800ec80 tx=0x7fd2980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd291ffb700 1 -- 172.21.15.121:0/1730465114 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29800c8f0 con 0x7fd29c0a6950 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd291ffb700 1 -- 172.21.15.121:0/1730465114 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29800ca50 con 0x7fd29c0a6950 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd294009ae0 con 0x7fd29c0a6950 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.238+0000 7fd291ffb700 1 -- 172.21.15.121:0/1730465114 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29800cdd0 con 0x7fd29c0a6950 2023-09-29T23:10:51.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.239+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c13fa40 con 0x7fd29c0a6950 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.239+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c071660 con 0x7fd29c0a6950 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.240+0000 7fd291ffb700 1 -- 172.21.15.121:0/1730465114 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd298013680 con 0x7fd29c0a6950 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.240+0000 7fd291ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.241+0000 7fd291ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.241+0000 7fd291ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.241+0000 7fd291ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.241+0000 7fd291ffb700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd280098780 0x7fd28009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.241+0000 7fd2a1347700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd280098780 0x7fd28009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.241+0000 7fd291ffb700 1 -- 172.21.15.121:0/1730465114 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fd2980c0b50 con 0x7fd29c0a6950 2023-09-29T23:10:51.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.241+0000 7fd2a1347700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd280098780 0x7fd28009ac30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd29c146b90 tx=0x7fd28c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:51.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.245+0000 7fd291ffb700 1 -- 172.21.15.121:0/1730465114 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd29808e200 con 0x7fd29c0a6950 2023-09-29T23:10:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.468+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd29c13ea50 con 0x7fd29c0a6950 2023-09-29T23:10:51.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.468+0000 7fd291ffb700 1 -- 172.21.15.121:0/1730465114 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd298091820 con 0x7fd29c0a6950 2023-09-29T23:10:51.469 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:51.469 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 10 mgrc shutdown 2023-09-29T23:10:51.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd280098780 msgr2=0x7fd28009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd280098780 0x7fd28009ac30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd29c146b90 tx=0x7fd28c009040 comp rx=0 tx=0).stop 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 msgr2=0x7fd29c143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c143bd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd29800ec80 tx=0x7fd2980091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 shutdown_connections 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fd280098780 0x7fd28009ac30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd29c1458e0 0x7fd29c0af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd29c0a6950 0x7fd29c143bd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 --2- 172.21.15.121:0/1730465114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd29c0a73b0 0x7fd29c0af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.470+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 >> 172.21.15.121:0/1730465114 conn(0x7fd29c06c010 msgr2=0x7fd29c0a4480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.471+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 shutdown_connections 2023-09-29T23:10:51.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:51.471+0000 7fd2a3dac700 1 -- 172.21.15.121:0/1730465114 wait complete. 2023-09-29T23:10:54.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:54.612 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.611+0000 7fed8377e700 1 Processor -- start 2023-09-29T23:10:54.612 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.611+0000 7fed8377e700 1 -- start start 2023-09-29T23:10:54.614 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.612+0000 7fed8377e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c06aec0 0x7fed7c09e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:54.614 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.612+0000 7fed8377e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed7c13e680 0x7fed7c13eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:54.614 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.612+0000 7fed8377e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c06b290 con 0x7fed7c0a6fe0 2023-09-29T23:10:54.614 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.612+0000 7fed8377e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7c06f020 con 0x7fed7c13e680 2023-09-29T23:10:54.614 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.612+0000 7fed8377e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c13f310 con 0x7fed7c06aec0 2023-09-29T23:10:54.614 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.612+0000 7fed81d1b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed7c13e680 0x7fed7c13eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed80d19700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fed7c0a6fe0 0x7fed7c06e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:35984/0 (socket says 172.21.15.121:35984) 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed8151a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c06aec0 0x7fed7c09e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed80d19700 1 -- 172.21.15.121:0/774665812 learned_addr learned my addr 172.21.15.121:0/774665812 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed8151a700 1 -- 172.21.15.121:0/774665812 >> v2:172.21.15.121:3300/0 conn(0x7fed7c13e680 msgr2=0x7fed7c13eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed8151a700 1 --2- 172.21.15.121:0/774665812 >> v2:172.21.15.121:3300/0 conn(0x7fed7c13e680 0x7fed7c13eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed8151a700 1 -- 172.21.15.121:0/774665812 >> v1:172.21.15.121:6789/0 conn(0x7fed7c0a6fe0 legacy=0x7fed7c06e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed8151a700 1 -- 172.21.15.121:0/774665812 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c06e110 con 0x7fed7c06aec0 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed81d1b700 1 --2- 172.21.15.121:0/774665812 >> v2:172.21.15.121:3300/0 conn(0x7fed7c13e680 0x7fed7c13eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed8151a700 1 --2- 172.21.15.121:0/774665812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c06aec0 0x7fed7c09e170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fed7000dd40 tx=0x7fed7000d630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94efc1af3d6d2f8c server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed6ffff700 1 -- 172.21.15.121:0/774665812 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed70012070 con 0x7fed7c06aec0 2023-09-29T23:10:54.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed6ffff700 1 -- 172.21.15.121:0/774665812 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7000d2e0 con 0x7fed7c06aec0 2023-09-29T23:10:54.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.613+0000 7fed6ffff700 1 -- 172.21.15.121:0/774665812 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7000dea0 con 0x7fed7c06aec0 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 -- 172.21.15.121:0/774665812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c06aec0 msgr2=0x7fed7c09e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 --2- 172.21.15.121:0/774665812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c06aec0 0x7fed7c09e170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fed7000dd40 tx=0x7fed7000d630 comp rx=0 tx=0).stop 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 -- 172.21.15.121:0/774665812 shutdown_connections 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 --2- 172.21.15.121:0/774665812 >> v2:172.21.15.121:3300/0 conn(0x7fed7c13e680 0x7fed7c13eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 --2- 172.21.15.121:0/774665812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c06aec0 0x7fed7c09e170 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 -- 172.21.15.121:0/774665812 >> 172.21.15.121:0/774665812 conn(0x7fed7c067440 msgr2=0x7fed7c067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 -- 172.21.15.121:0/774665812 shutdown_connections 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.614+0000 7fed8377e700 1 -- 172.21.15.121:0/774665812 wait complete. 2023-09-29T23:10:54.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.615+0000 7fed8377e700 1 Processor -- start 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.615+0000 7fed8377e700 1 -- start start 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.615+0000 7fed8377e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c06aec0 0x7fed7c0a3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.615+0000 7fed8377e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7c0a6fe0 0x7fed7c09fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.615+0000 7fed8377e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c13e680 0x7fed7c14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8377e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c14d4f0 con 0x7fed7c06aec0 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8377e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0a4660 con 0x7fed7c0a6fe0 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8377e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c14cb90 con 0x7fed7c13e680 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8151a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c06aec0 0x7fed7c0a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8151a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c06aec0 0x7fed7c0a3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53978/0 (socket says 172.21.15.121:53978) 2023-09-29T23:10:54.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8151a700 1 -- 172.21.15.121:0/435769687 learned_addr learned my addr 172.21.15.121:0/435769687 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed81d1b700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c13e680 0x7fed7c14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed80d19700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7c0a6fe0 0x7fed7c09fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed81d1b700 1 -- 172.21.15.121:0/435769687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7c0a6fe0 msgr2=0x7fed7c09fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed81d1b700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7c0a6fe0 0x7fed7c09fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed81d1b700 1 -- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c06aec0 msgr2=0x7fed7c0a3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed81d1b700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c06aec0 0x7fed7c0a3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed81d1b700 1 -- 172.21.15.121:0/435769687 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed74009a20 con 0x7fed7c13e680 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8151a700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c06aec0 0x7fed7c0a3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed80d19700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7c0a6fe0 0x7fed7c09fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed81d1b700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c13e680 0x7fed7c14c1b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fed74009e70 tx=0x7fed7400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:54.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed6d7fa700 1 -- 172.21.15.121:0/435769687 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7400cb50 con 0x7fed7c13e680 2023-09-29T23:10:54.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed6d7fa700 1 -- 172.21.15.121:0/435769687 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7400f040 con 0x7fed7c13e680 2023-09-29T23:10:54.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed6d7fa700 1 -- 172.21.15.121:0/435769687 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74014480 con 0x7fed7c13e680 2023-09-29T23:10:54.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7000d8d0 con 0x7fed7c13e680 2023-09-29T23:10:54.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.616+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c0a0280 con 0x7fed7c13e680 2023-09-29T23:10:54.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.617+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7c071660 con 0x7fed7c13e680 2023-09-29T23:10:54.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.618+0000 7fed6d7fa700 1 -- 172.21.15.121:0/435769687 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fed74003020 con 0x7fed7c13e680 2023-09-29T23:10:54.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.618+0000 7fed6d7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:54.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.619+0000 7fed6d7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:54.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.619+0000 7fed6d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:54.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.619+0000 7fed6d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:54.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.619+0000 7fed6d7fa700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fed60098780 0x7fed6009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:54.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.619+0000 7fed8151a700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fed60098780 0x7fed6009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:54.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.619+0000 7fed6d7fa700 1 -- 172.21.15.121:0/435769687 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fed740c0cc0 con 0x7fed7c13e680 2023-09-29T23:10:54.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.619+0000 7fed8151a700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fed60098780 0x7fed6009ac30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fed7000dd10 tx=0x7fed7001c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:54.624 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.623+0000 7fed6d7fa700 1 -- 172.21.15.121:0/435769687 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed7408e370 con 0x7fed7c13e680 2023-09-29T23:10:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.841+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fed7c0698b0 con 0x7fed7c13e680 2023-09-29T23:10:54.842 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.841+0000 7fed6d7fa700 1 -- 172.21.15.121:0/435769687 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fed74091990 con 0x7fed7c13e680 2023-09-29T23:10:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:54.842 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:54.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 10 mgrc shutdown 2023-09-29T23:10:54.844 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fed60098780 msgr2=0x7fed6009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fed60098780 0x7fed6009ac30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fed7000dd10 tx=0x7fed7001c000 comp rx=0 tx=0).stop 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c13e680 msgr2=0x7fed7c14c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c13e680 0x7fed7c14c1b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fed74009e70 tx=0x7fed7400b040 comp rx=0 tx=0).stop 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 shutdown_connections 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fed60098780 0x7fed6009ac30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed7c13e680 0x7fed7c14c1b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c06aec0 0x7fed7c0a3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 --2- 172.21.15.121:0/435769687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7c0a6fe0 0x7fed7c09fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 >> 172.21.15.121:0/435769687 conn(0x7fed7c067440 msgr2=0x7fed7c06e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 shutdown_connections 2023-09-29T23:10:54.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:54.843+0000 7fed8377e700 1 -- 172.21.15.121:0/435769687 wait complete. 2023-09-29T23:10:57.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:10:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.985+0000 7fe47f779700 1 Processor -- start 2023-09-29T23:10:57.986 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.985+0000 7fe47f779700 1 -- start start 2023-09-29T23:10:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47f779700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780b3cf0 0x7fe478146900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:57.987 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47f779700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe4780a2180 0x7fe478148e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47f779700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780a0620 con 0x7fe4780630e0 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47f779700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4780a08e0 con 0x7fe4780a2180 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47f779700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809c8d0 con 0x7fe4780b3cf0 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47cd14700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780b3cf0 0x7fe478146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47d515700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe4780630e0 0x7fe47809c1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53100/0 (socket says 172.21.15.121:53100) 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47d515700 1 -- 172.21.15.121:0/3268271008 learned_addr learned my addr 172.21.15.121:0/3268271008 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47dd16700 1 --2- 172.21.15.121:0/3268271008 >> v2:172.21.15.123:3300/0 conn(0x7fe4780a2180 0x7fe478148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe46ffff700 1 -- 172.21.15.121:0/3268271008 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913015841 0 0) 0x7fe4780a0620 con 0x7fe4780630e0 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47cd14700 1 -- 172.21.15.121:0/3268271008 >> v2:172.21.15.123:3300/0 conn(0x7fe4780a2180 msgr2=0x7fe478148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47cd14700 1 --2- 172.21.15.121:0/3268271008 >> v2:172.21.15.123:3300/0 conn(0x7fe4780a2180 0x7fe478148e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47cd14700 1 -- 172.21.15.121:0/3268271008 >> v1:172.21.15.121:6789/0 conn(0x7fe4780630e0 legacy=0x7fe47809c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47cd14700 1 -- 172.21.15.121:0/3268271008 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780a62d0 con 0x7fe4780b3cf0 2023-09-29T23:10:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe47cd14700 1 --2- 172.21.15.121:0/3268271008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780b3cf0 0x7fe478146900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe4680098e0 tx=0x7fe4680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82dbef7a5f20f909 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe46ffff700 1 -- 172.21.15.121:0/3268271008 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468009e80 con 0x7fe4780b3cf0 2023-09-29T23:10:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe46ffff700 1 -- 172.21.15.121:0/3268271008 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46800b860 con 0x7fe4780b3cf0 2023-09-29T23:10:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.986+0000 7fe46ffff700 1 -- 172.21.15.121:0/3268271008 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46800bbe0 con 0x7fe4780b3cf0 2023-09-29T23:10:57.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 -- 172.21.15.121:0/3268271008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780b3cf0 msgr2=0x7fe478146900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 --2- 172.21.15.121:0/3268271008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780b3cf0 0x7fe478146900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe4680098e0 tx=0x7fe4680091a0 comp rx=0 tx=0).stop 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 -- 172.21.15.121:0/3268271008 shutdown_connections 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 --2- 172.21.15.121:0/3268271008 >> v2:172.21.15.123:3300/0 conn(0x7fe4780a2180 0x7fe478148e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 --2- 172.21.15.121:0/3268271008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780b3cf0 0x7fe478146900 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 -- 172.21.15.121:0/3268271008 >> 172.21.15.121:0/3268271008 conn(0x7fe478069e90 msgr2=0x7fe4780b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 -- 172.21.15.121:0/3268271008 shutdown_connections 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.987+0000 7fe47f779700 1 -- 172.21.15.121:0/3268271008 wait complete. 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.988+0000 7fe47f779700 1 Processor -- start 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.988+0000 7fe47f779700 1 -- start start 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47f779700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780630e0 0x7fe478140bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:57.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47f779700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4780a2180 0x7fe4780a7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47f779700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780b3cf0 0x7fe47809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47f779700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809eb10 con 0x7fe4780b3cf0 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47f779700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe478099f30 con 0x7fe4780a2180 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47f779700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809a1f0 con 0x7fe4780630e0 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47d515700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780630e0 0x7fe478140bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47d515700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780630e0 0x7fe478140bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37708/0 (socket says 172.21.15.121:37708) 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47d515700 1 -- 172.21.15.121:0/3621615615 learned_addr learned my addr 172.21.15.121:0/3621615615 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.989+0000 7fe47dd16700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780b3cf0 0x7fe47809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47cd14700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4780a2180 0x7fe4780a7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47d515700 1 -- 172.21.15.121:0/3621615615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4780a2180 msgr2=0x7fe4780a7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47d515700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4780a2180 0x7fe4780a7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:57.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47d515700 1 -- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780b3cf0 msgr2=0x7fe47809db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47d515700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780b3cf0 0x7fe47809db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47d515700 1 -- 172.21.15.121:0/3621615615 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468009420 con 0x7fe4780630e0 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47d515700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780630e0 0x7fe478140bd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c7e0 tx=0x7fe474007c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe46d7fa700 1 -- 172.21.15.121:0/3621615615 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740072e0 con 0x7fe4780630e0 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe46d7fa700 1 -- 172.21.15.121:0/3621615615 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe474007460 con 0x7fe4780630e0 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe46d7fa700 1 -- 172.21.15.121:0/3621615615 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740057f0 con 0x7fe4780630e0 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47809a4b0 con 0x7fe4780630e0 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.990+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe478141a00 con 0x7fe4780630e0 2023-09-29T23:10:57.994 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.991+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47806ab60 con 0x7fe4780630e0 2023-09-29T23:10:57.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.992+0000 7fe46d7fa700 1 -- 172.21.15.121:0/3621615615 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe474005b60 con 0x7fe4780630e0 2023-09-29T23:10:57.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.992+0000 7fe46d7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:10:57.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.992+0000 7fe46d7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:10:57.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.992+0000 7fe46d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.992+0000 7fe46d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:10:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.992+0000 7fe46d7fa700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe45c098780 0x7fe45c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:10:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.993+0000 7fe47cd14700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe45c098780 0x7fe45c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:10:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.993+0000 7fe46d7fa700 1 -- 172.21.15.121:0/3621615615 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fe474001e40 con 0x7fe4780630e0 2023-09-29T23:10:57.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.993+0000 7fe47cd14700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe45c098780 0x7fe45c09ac30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe4680098e0 tx=0x7fe468017000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:10:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:57.997+0000 7fe46d7fa700 1 -- 172.21.15.121:0/3621615615 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe47404b5a0 con 0x7fe4780630e0 2023-09-29T23:10:58.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.216+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe478071660 con 0x7fe4780630e0 2023-09-29T23:10:58.217 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.216+0000 7fe46d7fa700 1 -- 172.21.15.121:0/3621615615 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe47408d050 con 0x7fe4780630e0 2023-09-29T23:10:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:10:58.217 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:10:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 10 mgrc shutdown 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe45c098780 msgr2=0x7fe45c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe45c098780 0x7fe45c09ac30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe4680098e0 tx=0x7fe468017000 comp rx=0 tx=0).stop 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780630e0 msgr2=0x7fe478140bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780630e0 0x7fe478140bd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c7e0 tx=0x7fe474007c80 comp rx=0 tx=0).stop 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 shutdown_connections 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe45c098780 0x7fe45c09ac30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780b3cf0 0x7fe47809db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4780630e0 0x7fe478140bd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:58.220 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 --2- 172.21.15.121:0/3621615615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4780a2180 0x7fe4780a7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:10:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 >> 172.21.15.121:0/3621615615 conn(0x7fe478069e90 msgr2=0x7fe4780b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:10:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 shutdown_connections 2023-09-29T23:10:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:10:58.218+0000 7fe47f779700 1 -- 172.21.15.121:0/3621615615 wait complete. 2023-09-29T23:11:01.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:01.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.360+0000 7f49681bf700 1 Processor -- start 2023-09-29T23:11:01.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.360+0000 7f49681bf700 1 -- start start 2023-09-29T23:11:01.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f49681bf700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f496009b770 0x7f49600630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:01.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f49681bf700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496014b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:01.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f49681bf700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4960141810 con 0x7f4960141440 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f49681bf700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496009a7f0 con 0x7f496009b770 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f49681bf700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49600a4df0 con 0x7f49600b3cf0 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f496675c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f4965f5b700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4960141440 0x7f49600a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:53104/0 (socket says 172.21.15.121:53104) 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f4965f5b700 1 -- 172.21.15.121:0/1809504969 learned_addr learned my addr 172.21.15.121:0/1809504969 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f496575a700 1 --2- 172.21.15.121:0/1809504969 >> v2:172.21.15.123:3300/0 conn(0x7f496009b770 0x7f49600630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f4964f59700 1 -- 172.21.15.121:0/1809504969 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862531900 0 0) 0x7f4960141810 con 0x7f4960141440 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.361+0000 7f4964f59700 1 -- 172.21.15.121:0/1809504969 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c003570 con 0x7f4960141440 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f496675c700 1 -- 172.21.15.121:0/1809504969 >> v2:172.21.15.123:3300/0 conn(0x7f496009b770 msgr2=0x7f49600630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:01.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f496675c700 1 --2- 172.21.15.121:0/1809504969 >> v2:172.21.15.123:3300/0 conn(0x7f496009b770 0x7f49600630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f496675c700 1 -- 172.21.15.121:0/1809504969 >> v1:172.21.15.121:6789/0 conn(0x7f4960141440 legacy=0x7f49600a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f496675c700 1 -- 172.21.15.121:0/1809504969 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49600a50b0 con 0x7f49600b3cf0 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f496675c700 1 --2- 172.21.15.121:0/1809504969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496014b190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4958009e60 tx=0x7f4958009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deddd198ee635652 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f4964f59700 1 -- 172.21.15.121:0/1809504969 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495800e070 con 0x7f49600b3cf0 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f4964f59700 1 -- 172.21.15.121:0/1809504969 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495800bab0 con 0x7f49600b3cf0 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f4964f59700 1 -- 172.21.15.121:0/1809504969 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495800be70 con 0x7f49600b3cf0 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f49681bf700 1 -- 172.21.15.121:0/1809504969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 msgr2=0x7f496014b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f49681bf700 1 --2- 172.21.15.121:0/1809504969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496014b190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4958009e60 tx=0x7f4958009780 comp rx=0 tx=0).stop 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f49681bf700 1 -- 172.21.15.121:0/1809504969 shutdown_connections 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f49681bf700 1 --2- 172.21.15.121:0/1809504969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496014b190 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f49681bf700 1 --2- 172.21.15.121:0/1809504969 >> v2:172.21.15.123:3300/0 conn(0x7f496009b770 0x7f49600630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.362+0000 7f49681bf700 1 -- 172.21.15.121:0/1809504969 >> 172.21.15.121:0/1809504969 conn(0x7f4960069e90 msgr2=0x7f49600b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:01.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.363+0000 7f49681bf700 1 -- 172.21.15.121:0/1809504969 shutdown_connections 2023-09-29T23:11:01.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.363+0000 7f49681bf700 1 -- 172.21.15.121:0/1809504969 wait complete. 2023-09-29T23:11:01.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.363+0000 7f49681bf700 1 Processor -- start 2023-09-29T23:11:01.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f49681bf700 1 -- start start 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f49681bf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496009b770 0x7f4960145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f49681bf700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496009d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f49681bf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4960141440 0x7f4960153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f49681bf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496009f280 con 0x7f4960141440 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f49681bf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496009e360 con 0x7f496009b770 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f49681bf700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496006da90 con 0x7f49600b3cf0 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496009d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496675c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4960141440 0x7f4960153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496009d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37732/0 (socket says 172.21.15.121:37732) 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 -- 172.21.15.121:0/1296109392 learned_addr learned my addr 172.21.15.121:0/1296109392 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:01.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f4965f5b700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496009b770 0x7f4960145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 -- 172.21.15.121:0/1296109392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496009b770 msgr2=0x7f4960145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496009b770 0x7f4960145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 -- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4960141440 msgr2=0x7f4960153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4960141440 0x7f4960153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496575a700 1 -- 172.21.15.121:0/1296109392 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4958009a20 con 0x7f49600b3cf0 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f496675c700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4960141440 0x7f4960153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.364+0000 7f4965f5b700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496009b770 0x7f4960145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.365+0000 7f496575a700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496009d5f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f495c00ec50 tx=0x7f495c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.365+0000 7f4951ffb700 1 -- 172.21.15.121:0/1296109392 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c00c850 con 0x7f49600b3cf0 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.365+0000 7f4951ffb700 1 -- 172.21.15.121:0/1296109392 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495c00c9b0 con 0x7f49600b3cf0 2023-09-29T23:11:01.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.365+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496006dd50 con 0x7f49600b3cf0 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.365+0000 7f4951ffb700 1 -- 172.21.15.121:0/1296109392 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c00cd70 con 0x7f49600b3cf0 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.365+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49601458a0 con 0x7f49600b3cf0 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.366+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496006ab60 con 0x7f49600b3cf0 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.367+0000 7f4951ffb700 1 -- 172.21.15.121:0/1296109392 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f495c013680 con 0x7f49600b3cf0 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.367+0000 7f4951ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.367+0000 7f4951ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.367+0000 7f4951ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.367+0000 7f4951ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.367+0000 7f4951ffb700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f49440987b0 0x7f494409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.368+0000 7f4965f5b700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f49440987b0 0x7f494409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.368+0000 7f4951ffb700 1 -- 172.21.15.121:0/1296109392 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f495c0c0ba0 con 0x7f49600b3cf0 2023-09-29T23:11:01.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.368+0000 7f4965f5b700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f49440987b0 0x7f494409ac60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4954001360 tx=0x7f4954009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:01.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.371+0000 7f4951ffb700 1 -- 172.21.15.121:0/1296109392 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f495c08e1c0 con 0x7f49600b3cf0 2023-09-29T23:11:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.593+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4960071660 con 0x7f49600b3cf0 2023-09-29T23:11:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.593+0000 7f4951ffb700 1 -- 172.21.15.121:0/1296109392 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f495c017100 con 0x7f49600b3cf0 2023-09-29T23:11:01.595 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:01.595 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.595+0000 7f49681bf700 10 mgrc shutdown 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f49440987b0 msgr2=0x7f494409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f49440987b0 0x7f494409ac60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4954001360 tx=0x7f4954009040 comp rx=0 tx=0).stop 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 msgr2=0x7f496009d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496009d5f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f495c00ec50 tx=0x7f495c0091a0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 shutdown_connections 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f49440987b0 0x7f494409ac60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4960141440 0x7f4960153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496009b770 0x7f4960145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 --2- 172.21.15.121:0/1296109392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49600b3cf0 0x7f496009d5f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 >> 172.21.15.121:0/1296109392 conn(0x7f4960069e90 msgr2=0x7f49601403d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 shutdown_connections 2023-09-29T23:11:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:01.596+0000 7f49681bf700 1 -- 172.21.15.121:0/1296109392 wait complete. 2023-09-29T23:11:04.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 Processor -- start 2023-09-29T23:11:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 -- start start 2023-09-29T23:11:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb8dc141440 0x7fb8dc0a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc09b770 0x7fb8dc0630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb8dc0b3cf0 0x7fb8dc143df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8dc141810 con 0x7fb8dc141440 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8dc09a7f0 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e4165700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0a4df0 con 0x7fb8dc09b770 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc09b770 0x7fb8dc0630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1f01700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb8dc141440 0x7fb8dc0a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc09b770 0x7fb8dc0630e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37742/0 (socket says 172.21.15.121:37742) 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 -- 172.21.15.121:0/2144053551 learned_addr learned my addr 172.21.15.121:0/2144053551 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e2702700 1 --2- 172.21.15.121:0/2144053551 >> v2:172.21.15.123:3300/0 conn(0x7fb8dc0b3cf0 0x7fb8dc143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 -- 172.21.15.121:0/2144053551 >> v2:172.21.15.123:3300/0 conn(0x7fb8dc0b3cf0 msgr2=0x7fb8dc143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 --2- 172.21.15.121:0/2144053551 >> v2:172.21.15.123:3300/0 conn(0x7fb8dc0b3cf0 0x7fb8dc143df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 -- 172.21.15.121:0/2144053551 >> v2:172.21.15.121:3300/0 conn(0x7fb8dc141440 msgr2=0x7fb8dc0a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 --2- 172.21.15.121:0/2144053551 >> v2:172.21.15.121:3300/0 conn(0x7fb8dc141440 0x7fb8dc0a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 -- 172.21.15.121:0/2144053551 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc0a50b0 con 0x7fb8dc09b770 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1f01700 1 --2- 172.21.15.121:0/2144053551 >> v2:172.21.15.121:3300/0 conn(0x7fb8dc141440 0x7fb8dc0a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e2702700 1 --2- 172.21.15.121:0/2144053551 >> v2:172.21.15.123:3300/0 conn(0x7fb8dc0b3cf0 0x7fb8dc143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e1700700 1 --2- 172.21.15.121:0/2144053551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc09b770 0x7fb8dc0630e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc06b1b0 tx=0x7fb8cc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f096ab9d2562bb4d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e0eff700 1 -- 172.21.15.121:0/2144053551 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc009e70 con 0x7fb8dc09b770 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e0eff700 1 -- 172.21.15.121:0/2144053551 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc00b860 con 0x7fb8dc09b770 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.738+0000 7fb8e0eff700 1 -- 172.21.15.121:0/2144053551 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc00bc20 con 0x7fb8dc09b770 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 -- 172.21.15.121:0/2144053551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc09b770 msgr2=0x7fb8dc0630e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 --2- 172.21.15.121:0/2144053551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc09b770 0x7fb8dc0630e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc06b1b0 tx=0x7fb8cc009250 comp rx=0 tx=0).stop 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 -- 172.21.15.121:0/2144053551 shutdown_connections 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 --2- 172.21.15.121:0/2144053551 >> v2:172.21.15.123:3300/0 conn(0x7fb8dc0b3cf0 0x7fb8dc143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 --2- 172.21.15.121:0/2144053551 >> v2:172.21.15.121:3300/0 conn(0x7fb8dc141440 0x7fb8dc0a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 --2- 172.21.15.121:0/2144053551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc09b770 0x7fb8dc0630e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 -- 172.21.15.121:0/2144053551 >> 172.21.15.121:0/2144053551 conn(0x7fb8dc069e90 msgr2=0x7fb8dc0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 -- 172.21.15.121:0/2144053551 shutdown_connections 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.739+0000 7fb8e4165700 1 -- 172.21.15.121:0/2144053551 wait complete. 2023-09-29T23:11:04.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 Processor -- start 2023-09-29T23:11:04.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 -- start start 2023-09-29T23:11:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8dc09b770 0x7fb8dc0b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8dc0b3cf0 0x7fb8dc0a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc141440 0x7fb8dc09ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc0a1460 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc09efc0 con 0x7fb8dc09b770 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.740+0000 7fb8e4165700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc09f1d0 con 0x7fb8dc141440 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8dc0b3cf0 0x7fb8dc0a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e2702700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc141440 0x7fb8dc09ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8dc0b3cf0 0x7fb8dc0a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50894/0 (socket says 172.21.15.121:50894) 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 -- 172.21.15.121:0/3504091503 learned_addr learned my addr 172.21.15.121:0/3504091503 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1f01700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8dc09b770 0x7fb8dc0b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 -- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc141440 msgr2=0x7fb8dc09ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc141440 0x7fb8dc09ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 -- 172.21.15.121:0/3504091503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8dc09b770 msgr2=0x7fb8dc0b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8dc09b770 0x7fb8dc0b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 -- 172.21.15.121:0/3504091503 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d80093c0 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e2702700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc141440 0x7fb8dc09ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1f01700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8dc09b770 0x7fb8dc0b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e1700700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8dc0b3cf0 0x7fb8dc0a0370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0a4f30 tx=0x7fb8cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8d1ffb700 1 -- 172.21.15.121:0/3504091503 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc01f070 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8d1ffb700 1 -- 172.21.15.121:0/3504091503 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc006a90 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8d1ffb700 1 -- 172.21.15.121:0/3504091503 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc00b9d0 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc0094f0 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.741+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc09e690 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.742+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8dc06ab60 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.743+0000 7fb8d1ffb700 1 -- 172.21.15.121:0/3504091503 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb8cc023070 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.743+0000 7fb8d1ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.744+0000 7fb8d1ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.744+0000 7fb8d1ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.744+0000 7fb8d1ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.744+0000 7fb8d1ffb700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8c00987b0 0x7fb8c009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.744+0000 7fb8e1f01700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8c00987b0 0x7fb8c009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.744+0000 7fb8d1ffb700 1 -- 172.21.15.121:0/3504091503 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fb8cc0c11c0 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.744+0000 7fb8e1f01700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8c00987b0 0x7fb8c009ac60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009390 tx=0x7fb8d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.748+0000 7fb8d1ffb700 1 -- 172.21.15.121:0/3504091503 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8cc08e870 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.969+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8dc09e850 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.970 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.970+0000 7fb8d1ffb700 1 -- 172.21.15.121:0/3504091503 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb8cc091e90 con 0x7fb8dc0b3cf0 2023-09-29T23:11:04.971 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:04.971 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 10 mgrc shutdown 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8c00987b0 msgr2=0x7fb8c009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8c00987b0 0x7fb8c009ac60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009390 tx=0x7fb8d8008040 comp rx=0 tx=0).stop 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8dc0b3cf0 msgr2=0x7fb8dc0a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8dc0b3cf0 0x7fb8dc0a0370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0a4f30 tx=0x7fb8cc017000 comp rx=0 tx=0).stop 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 shutdown_connections 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fb8c00987b0 0x7fb8c009ac60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8dc141440 0x7fb8dc09ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8dc09b770 0x7fb8dc0b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.973 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 --2- 172.21.15.121:0/3504091503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8dc0b3cf0 0x7fb8dc0a0370 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 >> 172.21.15.121:0/3504091503 conn(0x7fb8dc069e90 msgr2=0x7fb8dc0a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 shutdown_connections 2023-09-29T23:11:04.974 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:04.972+0000 7fb8e4165700 1 -- 172.21.15.121:0/3504091503 wait complete. 2023-09-29T23:11:07.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:08.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.112+0000 7fed134dc700 1 Processor -- start 2023-09-29T23:11:08.112 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.112+0000 7fed134dc700 1 -- start start 2023-09-29T23:11:08.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.112+0000 7fed134dc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fed0c0a6a00 0x7fed0c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:08.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.112+0000 7fed134dc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed0c0a5510 0x7fed0c147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:08.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.112+0000 7fed134dc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c141700 con 0x7fed0c0a6630 2023-09-29T23:11:08.114 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.112+0000 7fed134dc700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c0b4370 con 0x7fed0c0a5510 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.112+0000 7fed134dc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c06ee60 con 0x7fed0c0a6a00 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed0c0a5510 0x7fed0c147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed0c0a5510 0x7fed0c147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53144/0 (socket says 172.21.15.121:53144) 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 -- 172.21.15.121:0/2652838302 learned_addr learned my addr 172.21.15.121:0/2652838302 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed10a77700 1 --2- 172.21.15.121:0/2652838302 >> v2:172.21.15.123:3300/0 conn(0x7fed0c0a6a00 0x7fed0c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 -- 172.21.15.121:0/2652838302 >> v2:172.21.15.123:3300/0 conn(0x7fed0c0a6a00 msgr2=0x7fed0c0a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 --2- 172.21.15.121:0/2652838302 >> v2:172.21.15.123:3300/0 conn(0x7fed0c0a6a00 0x7fed0c0a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 -- 172.21.15.121:0/2652838302 >> v1:172.21.15.123:6789/0 conn(0x7fed0c0a6630 legacy=0x7fed0c140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 -- 172.21.15.121:0/2652838302 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c140980 con 0x7fed0c0a5510 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed11a79700 1 --2- 172.21.15.121:0/2652838302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a5510 0x7fed0c147ba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0a6e40 tx=0x7fed04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab12b5fc06bc881 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed03fff700 1 -- 172.21.15.121:0/2652838302 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0400e070 con 0x7fed0c0a5510 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed03fff700 1 -- 172.21.15.121:0/2652838302 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0400ba50 con 0x7fed0c0a5510 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.113+0000 7fed03fff700 1 -- 172.21.15.121:0/2652838302 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040092e0 con 0x7fed0c0a5510 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 -- 172.21.15.121:0/2652838302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a5510 msgr2=0x7fed0c147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 --2- 172.21.15.121:0/2652838302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a5510 0x7fed0c147ba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0a6e40 tx=0x7fed04009840 comp rx=0 tx=0).stop 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 -- 172.21.15.121:0/2652838302 shutdown_connections 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 --2- 172.21.15.121:0/2652838302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a5510 0x7fed0c147ba0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 --2- 172.21.15.121:0/2652838302 >> v2:172.21.15.123:3300/0 conn(0x7fed0c0a6a00 0x7fed0c0a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 -- 172.21.15.121:0/2652838302 >> 172.21.15.121:0/2652838302 conn(0x7fed0c06c010 msgr2=0x7fed0c06c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 -- 172.21.15.121:0/2652838302 shutdown_connections 2023-09-29T23:11:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.114+0000 7fed134dc700 1 -- 172.21.15.121:0/2652838302 wait complete. 2023-09-29T23:11:08.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.115+0000 7fed134dc700 1 Processor -- start 2023-09-29T23:11:08.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.115+0000 7fed134dc700 1 -- start start 2023-09-29T23:11:08.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed134dc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed0c0a5510 0x7fed0c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:08.119 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed134dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a6630 0x7fed0c068410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed134dc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c0a6a00 0x7fed0c0a7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed134dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c14b7f0 con 0x7fed0c0a6630 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed134dc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0ac3c0 con 0x7fed0c0a6a00 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed134dc700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0a87f0 con 0x7fed0c0a5510 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed10a77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a6630 0x7fed0c068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed11278700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed0c0a5510 0x7fed0c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed10a77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a6630 0x7fed0c068410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53148/0 (socket says 172.21.15.121:53148) 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.116+0000 7fed10a77700 1 -- 172.21.15.121:0/2868094019 learned_addr learned my addr 172.21.15.121:0/2868094019 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11a79700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c0a6a00 0x7fed0c0a7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11278700 1 -- 172.21.15.121:0/2868094019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c0a6a00 msgr2=0x7fed0c0a7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11278700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c0a6a00 0x7fed0c0a7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.120 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11278700 1 -- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a6630 msgr2=0x7fed0c068410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11278700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a6630 0x7fed0c068410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11278700 1 -- 172.21.15.121:0/2868094019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed04009ae0 con 0x7fed0c0a5510 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11a79700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c0a6a00 0x7fed0c0a7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed10a77700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a6630 0x7fed0c068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed11278700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed0c0a5510 0x7fed0c0abe80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fed0800c590 tx=0x7fed08007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed017fa700 1 -- 172.21.15.121:0/2868094019 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08011590 con 0x7fed0c0a5510 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed017fa700 1 -- 172.21.15.121:0/2868094019 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed08011710 con 0x7fed0c0a5510 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c068950 con 0x7fed0c0a5510 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed017fa700 1 -- 172.21.15.121:0/2868094019 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08011ad0 con 0x7fed0c0a5510 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.117+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c0a8360 con 0x7fed0c0a5510 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.118+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c0a7b40 con 0x7fed0c0a5510 2023-09-29T23:11:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed017fa700 1 -- 172.21.15.121:0/2868094019 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fed0801b3b0 con 0x7fed0c0a5510 2023-09-29T23:11:08.122 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed017fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:08.123 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed017fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:08.123 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed017fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:08.124 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed017fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:08.124 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed017fa700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecf0098780 0x7fecf009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:08.124 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed10a77700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecf0098780 0x7fecf009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:08.124 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.120+0000 7fed017fa700 1 -- 172.21.15.121:0/2868094019 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fed080bfb20 con 0x7fed0c0a5510 2023-09-29T23:11:08.124 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.121+0000 7fed10a77700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecf0098780 0x7fecf009ac30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fed0c06ca40 tx=0x7fecfc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:08.124 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.124+0000 7fed017fa700 1 -- 172.21.15.121:0/2868094019 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed0808d1d0 con 0x7fed0c0a5510 2023-09-29T23:11:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.342+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fed0c0a7b40 con 0x7fed0c0a5510 2023-09-29T23:11:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.342+0000 7fed017fa700 1 -- 172.21.15.121:0/2868094019 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fed0c0a7b40 con 0x7fed0c0a5510 2023-09-29T23:11:08.342 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:08.343 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.344+0000 7fed134dc700 10 mgrc shutdown 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecf0098780 msgr2=0x7fecf009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecf0098780 0x7fecf009ac30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fed0c06ca40 tx=0x7fecfc008040 comp rx=0 tx=0).stop 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed0c0a5510 msgr2=0x7fed0c0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed0c0a5510 0x7fed0c0abe80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fed0800c590 tx=0x7fed08007ec0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 shutdown_connections 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fecf0098780 0x7fecf009ac30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c0a6a00 0x7fed0c0a7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed0c0a5510 0x7fed0c0abe80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.345 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 --2- 172.21.15.121:0/2868094019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed0c0a6630 0x7fed0c068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:08.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 >> 172.21.15.121:0/2868094019 conn(0x7fed0c06c010 msgr2=0x7fed0c139d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:08.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 shutdown_connections 2023-09-29T23:11:08.346 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:08.345+0000 7fed134dc700 1 -- 172.21.15.121:0/2868094019 wait complete. 2023-09-29T23:11:11.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:11.487 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.487+0000 7fc722e25700 1 Processor -- start 2023-09-29T23:11:11.487 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.487+0000 7fc722e25700 1 -- start start 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.487+0000 7fc722e25700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc71c0b5020 0x7fc71c09e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.487+0000 7fc722e25700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c0b53f0 con 0x7fc71c06b470 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc722e25700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71c06ae80 con 0x7fc71c13ede0 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc722e25700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71c13e2e0 con 0x7fc71c0b5020 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc720bc1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc71c0b5020 0x7fc71c09e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc71bfff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc71c06b470 0x7fc71c13e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33682/0 (socket says 172.21.15.121:33682) 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc720bc1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc71c0b5020 0x7fc71c09e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53154/0 (socket says 172.21.15.121:53154) 2023-09-29T23:11:11.489 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc71bfff700 1 -- 172.21.15.121:0/4139117320 learned_addr learned my addr 172.21.15.121:0/4139117320 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc720bc1700 1 -- 172.21.15.121:0/4139117320 >> v1:172.21.15.123:6789/0 conn(0x7fc71c13ede0 legacy=0x7fc71c148c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc720bc1700 1 -- 172.21.15.121:0/4139117320 >> v1:172.21.15.121:6789/0 conn(0x7fc71c06b470 legacy=0x7fc71c13e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc71b7fe700 1 -- 172.21.15.121:0/4139117320 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049945577 0 0) 0x7fc71c0b53f0 con 0x7fc71c06b470 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc720bc1700 1 -- 172.21.15.121:0/4139117320 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c06f390 con 0x7fc71c0b5020 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc720bc1700 1 --2- 172.21.15.121:0/4139117320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c0b5020 0x7fc71c09e260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7100098e0 tx=0x7fc7100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826c4b171dbccaf9 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc71b7fe700 1 -- 172.21.15.121:0/4139117320 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710009e80 con 0x7fc71c0b5020 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc71b7fe700 1 -- 172.21.15.121:0/4139117320 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71000b860 con 0x7fc71c0b5020 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.488+0000 7fc71b7fe700 1 -- 172.21.15.121:0/4139117320 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000bbe0 con 0x7fc71c0b5020 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.489+0000 7fc722e25700 1 -- 172.21.15.121:0/4139117320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c0b5020 msgr2=0x7fc71c09e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.489+0000 7fc722e25700 1 --2- 172.21.15.121:0/4139117320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c0b5020 0x7fc71c09e260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7100098e0 tx=0x7fc7100091a0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.490 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.489+0000 7fc722e25700 1 -- 172.21.15.121:0/4139117320 shutdown_connections 2023-09-29T23:11:11.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.489+0000 7fc722e25700 1 --2- 172.21.15.121:0/4139117320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c0b5020 0x7fc71c09e260 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.489+0000 7fc722e25700 1 -- 172.21.15.121:0/4139117320 >> 172.21.15.121:0/4139117320 conn(0x7fc71c06d0b0 msgr2=0x7fc71c06d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:11.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.489+0000 7fc722e25700 1 -- 172.21.15.121:0/4139117320 shutdown_connections 2023-09-29T23:11:11.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.489+0000 7fc722e25700 1 -- 172.21.15.121:0/4139117320 wait complete. 2023-09-29T23:11:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.490+0000 7fc722e25700 1 Processor -- start 2023-09-29T23:11:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.490+0000 7fc722e25700 1 -- start start 2023-09-29T23:11:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc722e25700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc71c06b470 0x7fc71c097760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc722e25700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc71c0b5020 0x7fc71c1417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc722e25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c13ede0 0x7fc71c14de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc722e25700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c14eb30 con 0x7fc71c13ede0 2023-09-29T23:11:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc722e25700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c14edf0 con 0x7fc71c0b5020 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc722e25700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c097ca0 con 0x7fc71c06b470 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc720bc1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc71c06b470 0x7fc71c097760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc7213c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c13ede0 0x7fc71c14de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc720bc1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc71c06b470 0x7fc71c097760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50882/0 (socket says 172.21.15.121:50882) 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc720bc1700 1 -- 172.21.15.121:0/981013578 learned_addr learned my addr 172.21.15.121:0/981013578 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.491+0000 7fc71bfff700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc71c0b5020 0x7fc71c1417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc720bc1700 1 -- 172.21.15.121:0/981013578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc71c0b5020 msgr2=0x7fc71c1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc720bc1700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc71c0b5020 0x7fc71c1417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc720bc1700 1 -- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c13ede0 msgr2=0x7fc71c14de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc720bc1700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c13ede0 0x7fc71c14de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc720bc1700 1 -- 172.21.15.121:0/981013578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710009420 con 0x7fc71c06b470 2023-09-29T23:11:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc720bc1700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc71c06b470 0x7fc71c097760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0b5780 tx=0x7fc710017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc718ff9700 1 -- 172.21.15.121:0/981013578 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71001f070 con 0x7fc71c06b470 2023-09-29T23:11:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc718ff9700 1 -- 172.21.15.121:0/981013578 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc710000c90 con 0x7fc71c06b470 2023-09-29T23:11:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc718ff9700 1 -- 172.21.15.121:0/981013578 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710025420 con 0x7fc71c06b470 2023-09-29T23:11:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c141cf0 con 0x7fc71c06b470 2023-09-29T23:11:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.492+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c1423b0 con 0x7fc71c06b470 2023-09-29T23:11:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.493+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c071660 con 0x7fc71c06b470 2023-09-29T23:11:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.494+0000 7fc718ff9700 1 -- 172.21.15.121:0/981013578 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc710023070 con 0x7fc71c06b470 2023-09-29T23:11:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.494+0000 7fc718ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.494+0000 7fc718ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.494+0000 7fc718ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.494+0000 7fc718ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.494+0000 7fc718ff9700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc700098780 0x7fc70009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:11.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.494+0000 7fc718ff9700 1 -- 172.21.15.121:0/981013578 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fc7100c1da0 con 0x7fc71c06b470 2023-09-29T23:11:11.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.495+0000 7fc71bfff700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc700098780 0x7fc70009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:11.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.495+0000 7fc71bfff700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc700098780 0x7fc70009ac30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0a4b40 tx=0x7fc708009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:11.497 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.498+0000 7fc718ff9700 1 -- 172.21.15.121:0/981013578 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc71008f450 con 0x7fc71c06b470 2023-09-29T23:11:11.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.719+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc71c09e7a0 con 0x7fc71c06b470 2023-09-29T23:11:11.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.719+0000 7fc718ff9700 1 -- 172.21.15.121:0/981013578 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc710092a70 con 0x7fc71c06b470 2023-09-29T23:11:11.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:11.720 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:11.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 10 mgrc shutdown 2023-09-29T23:11:11.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc700098780 msgr2=0x7fc70009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:11.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc700098780 0x7fc70009ac30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0a4b40 tx=0x7fc708009040 comp rx=0 tx=0).stop 2023-09-29T23:11:11.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc71c06b470 msgr2=0x7fc71c097760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:11.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc71c06b470 0x7fc71c097760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0b5780 tx=0x7fc710017000 comp rx=0 tx=0).stop 2023-09-29T23:11:11.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 shutdown_connections 2023-09-29T23:11:11.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fc700098780 0x7fc70009ac30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc71c13ede0 0x7fc71c14de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc71c06b470 0x7fc71c097760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 --2- 172.21.15.121:0/981013578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc71c0b5020 0x7fc71c1417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:11.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 >> 172.21.15.121:0/981013578 conn(0x7fc71c06d0b0 msgr2=0x7fc71c06d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:11.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 shutdown_connections 2023-09-29T23:11:11.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:11.721+0000 7fc722e25700 1 -- 172.21.15.121:0/981013578 wait complete. 2023-09-29T23:11:14.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:14.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.862+0000 7fe9bf0ff700 1 Processor -- start 2023-09-29T23:11:14.862 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.862+0000 7fe9bf0ff700 1 -- start start 2023-09-29T23:11:14.864 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bf0ff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe9b80a6630 0x7fe9b8140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bf0ff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe9b80a6a00 0x7fe9b80a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bf0ff700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b806ee60 con 0x7fe9b80a5510 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bf0ff700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b80b4370 con 0x7fe9b80a6630 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bf0ff700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b8140980 con 0x7fe9b80a6a00 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe9b80a6630 0x7fe9b8140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe9b80a6630 0x7fe9b8140ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53176/0 (socket says 172.21.15.121:53176) 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 -- 172.21.15.121:0/669481733 learned_addr learned my addr 172.21.15.121:0/669481733 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 -- 172.21.15.121:0/669481733 >> v2:172.21.15.123:3300/0 conn(0x7fe9b80a6a00 msgr2=0x7fe9b80a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9b7fff700 1 --2- 172.21.15.121:0/669481733 >> v2:172.21.15.123:3300/0 conn(0x7fe9b80a6a00 0x7fe9b80a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:14.865 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 --2- 172.21.15.121:0/669481733 >> v2:172.21.15.123:3300/0 conn(0x7fe9b80a6a00 0x7fe9b80a4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 -- 172.21.15.121:0/669481733 >> v1:172.21.15.123:6789/0 conn(0x7fe9b80a5510 legacy=0x7fe9b806cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 -- 172.21.15.121:0/669481733 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8140c40 con 0x7fe9b80a6630 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9bce9b700 1 --2- 172.21.15.121:0/669481733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6630 0x7fe9b8140ff0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80a6e40 tx=0x7fe9ac00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78d00e8363c062d server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9b77fe700 1 -- 172.21.15.121:0/669481733 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00dc10 con 0x7fe9b80a6630 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9b77fe700 1 -- 172.21.15.121:0/669481733 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac00dd70 con 0x7fe9b80a6630 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.863+0000 7fe9b77fe700 1 -- 172.21.15.121:0/669481733 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac0106c0 con 0x7fe9b80a6630 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/669481733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6630 msgr2=0x7fe9b8140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/669481733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6630 0x7fe9b8140ff0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80a6e40 tx=0x7fe9ac00b4f0 comp rx=0 tx=0).stop 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/669481733 shutdown_connections 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/669481733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6630 0x7fe9b8140ff0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:14.866 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/669481733 >> v2:172.21.15.123:3300/0 conn(0x7fe9b80a6a00 0x7fe9b80a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:14.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/669481733 >> 172.21.15.121:0/669481733 conn(0x7fe9b806c010 msgr2=0x7fe9b806c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:14.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/669481733 shutdown_connections 2023-09-29T23:11:14.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.864+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/669481733 wait complete. 2023-09-29T23:11:14.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.865+0000 7fe9bf0ff700 1 Processor -- start 2023-09-29T23:11:14.867 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.865+0000 7fe9bf0ff700 1 -- start start 2023-09-29T23:11:14.869 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bf0ff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9b80a5510 0x7fe9b80af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bf0ff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b80a6630 0x7fe9b8067de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bf0ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6a00 0x7fe9b8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bf0ff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b80689a0 con 0x7fe9b80a6a00 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bf0ff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b8068c60 con 0x7fe9b80a6630 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bf0ff700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b80ac020 con 0x7fe9b80a5510 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bce9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9b80a5510 0x7fe9b80af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6a00 0x7fe9b8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bce9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9b80a5510 0x7fe9b80af8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:50886/0 (socket says 172.21.15.121:50886) 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bce9b700 1 -- 172.21.15.121:0/1646489268 learned_addr learned my addr 172.21.15.121:0/1646489268 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6a00 0x7fe9b8068320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53182/0 (socket says 172.21.15.121:53182) 2023-09-29T23:11:14.870 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9b7fff700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b80a6630 0x7fe9b8067de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 -- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9b80a5510 msgr2=0x7fe9b80af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9b80a5510 0x7fe9b80af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 -- 172.21.15.121:0/1646489268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b80a6630 msgr2=0x7fe9b8067de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b80a6630 0x7fe9b8067de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 -- 172.21.15.121:0/1646489268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac00b790 con 0x7fe9b80a6a00 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bd69c700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6a00 0x7fe9b8068320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000bad0 tx=0x7fe9b0009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9b4ff9700 1 -- 172.21.15.121:0/1646489268 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000c7f0 con 0x7fe9b80a6a00 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9b4ff9700 1 -- 172.21.15.121:0/1646489268 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b000c950 con 0x7fe9b80a6a00 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b80accf0 con 0x7fe9b80a6a00 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.866+0000 7fe9b4ff9700 1 -- 172.21.15.121:0/1646489268 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000cd10 con 0x7fe9b80a6a00 2023-09-29T23:11:14.871 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.867+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b80affc0 con 0x7fe9b80a6a00 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.867+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b8071660 con 0x7fe9b80a6a00 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.868+0000 7fe9b4ff9700 1 -- 172.21.15.121:0/1646489268 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe9b0014620 con 0x7fe9b80a6a00 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.868+0000 7fe9b4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.869+0000 7fe9b4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.869+0000 7fe9b4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.869+0000 7fe9b4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.869+0000 7fe9b4ff9700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe99c0987b0 0x7fe99c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.869+0000 7fe9bce9b700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe99c0987b0 0x7fe99c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.869+0000 7fe9b4ff9700 1 -- 172.21.15.121:0/1646489268 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fe9b00c0330 con 0x7fe9b80a6a00 2023-09-29T23:11:14.872 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.869+0000 7fe9bce9b700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe99c0987b0 0x7fe99c09ac60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00b730 tx=0x7fe9ac01a000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:14.873 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:14.873+0000 7fe9b4ff9700 1 -- 172.21.15.121:0/1646489268 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9b008d950 con 0x7fe9b80a6a00 2023-09-29T23:11:15.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.094+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe9b80ac1e0 con 0x7fe9b80a6a00 2023-09-29T23:11:15.094 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.095+0000 7fe9b4ff9700 1 -- 172.21.15.121:0/1646489268 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe9b0090f70 con 0x7fe9b80a6a00 2023-09-29T23:11:15.095 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:15.095 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.096+0000 7fe9bf0ff700 10 mgrc shutdown 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.096+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe99c0987b0 msgr2=0x7fe99c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.096+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe99c0987b0 0x7fe99c09ac60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00b730 tx=0x7fe9ac01a000 comp rx=0 tx=0).stop 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.096+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6a00 msgr2=0x7fe9b8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.096+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6a00 0x7fe9b8068320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000bad0 tx=0x7fe9b0009760 comp rx=0 tx=0).stop 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 shutdown_connections 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe99c0987b0 0x7fe99c09ac60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9b80a6a00 0x7fe9b8068320 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9b80a5510 0x7fe9b80af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:15.097 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 --2- 172.21.15.121:0/1646489268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b80a6630 0x7fe9b8067de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:15.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 >> 172.21.15.121:0/1646489268 conn(0x7fe9b806c010 msgr2=0x7fe9b813ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:15.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 shutdown_connections 2023-09-29T23:11:15.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:15.097+0000 7fe9bf0ff700 1 -- 172.21.15.121:0/1646489268 wait complete. 2023-09-29T23:11:18.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.236+0000 7f991532c700 1 Processor -- start 2023-09-29T23:11:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.236+0000 7f991532c700 1 -- start start 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.236+0000 7f991532c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100b7560 0x7f99100a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f991532c700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99100a3650 con 0x7f99101402b0 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f991532c700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99100a97a0 con 0x7f99100a1fe0 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f991532c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99100a0fb0 con 0x7f99100b7560 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100b7560 0x7f99100a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990f7fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f99101402b0 0x7f991014aef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:57722/0 (socket says 172.21.15.121:57722) 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990f7fe700 1 -- 172.21.15.121:0/898571105 learned_addr learned my addr 172.21.15.121:0/898571105 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:11:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990dffb700 1 -- 172.21.15.121:0/898571105 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157941027 0 0) 0x7f99100a3650 con 0x7f99101402b0 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990dffb700 1 -- 172.21.15.121:0/898571105 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f8003570 con 0x7f99101402b0 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990effd700 1 -- 172.21.15.121:0/898571105 >> v1:172.21.15.123:6789/0 conn(0x7f99100a1fe0 legacy=0x7f991013fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990effd700 1 -- 172.21.15.121:0/898571105 >> v1:172.21.15.121:6789/0 conn(0x7f99101402b0 legacy=0x7f991014aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990effd700 1 -- 172.21.15.121:0/898571105 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99100a9a60 con 0x7f99100b7560 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990effd700 1 --2- 172.21.15.121:0/898571105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100b7560 0x7f99100a0a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9904009910 tx=0x7f99040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baca2f27f495e65a server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990dffb700 1 -- 172.21.15.121:0/898571105 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990400b680 con 0x7f99100b7560 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990dffb700 1 -- 172.21.15.121:0/898571105 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990400b7e0 con 0x7f99100b7560 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.237+0000 7f990dffb700 1 -- 172.21.15.121:0/898571105 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990400bba0 con 0x7f99100b7560 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.238+0000 7f991532c700 1 -- 172.21.15.121:0/898571105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100b7560 msgr2=0x7f99100a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.238+0000 7f991532c700 1 --2- 172.21.15.121:0/898571105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100b7560 0x7f99100a0a70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9904009910 tx=0x7f99040091a0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.238+0000 7f991532c700 1 -- 172.21.15.121:0/898571105 shutdown_connections 2023-09-29T23:11:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.238+0000 7f991532c700 1 --2- 172.21.15.121:0/898571105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100b7560 0x7f99100a0a70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.238+0000 7f991532c700 1 -- 172.21.15.121:0/898571105 >> 172.21.15.121:0/898571105 conn(0x7f9910068350 msgr2=0x7f99100b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.238+0000 7f991532c700 1 -- 172.21.15.121:0/898571105 shutdown_connections 2023-09-29T23:11:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.238+0000 7f991532c700 1 -- 172.21.15.121:0/898571105 wait complete. 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.239+0000 7f991532c700 1 Processor -- start 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.239+0000 7f991532c700 1 -- start start 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.239+0000 7f991532c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100a1fe0 0x7f9910096130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f991532c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99100b7560 0x7f991013bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f991532c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99101402b0 0x7f991013c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f991532c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991013ca70 con 0x7f99101402b0 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f991532c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99100a61f0 con 0x7f99100b7560 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f991532c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99100a63f0 con 0x7f99100a1fe0 2023-09-29T23:11:18.243 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100a1fe0 0x7f9910096130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100a1fe0 0x7f9910096130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34626/0 (socket says 172.21.15.121:34626) 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 -- 172.21.15.121:0/1091403019 learned_addr learned my addr 172.21.15.121:0/1091403019 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99101402b0 0x7f991013c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990e7fc700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99100b7560 0x7f991013bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 -- 172.21.15.121:0/1091403019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99100b7560 msgr2=0x7f991013bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99100b7560 0x7f991013bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 -- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99101402b0 msgr2=0x7f991013c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99101402b0 0x7f991013c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 -- 172.21.15.121:0/1091403019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9904009420 con 0x7f99100a1fe0 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990f7fe700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99101402b0 0x7f991013c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f990effd700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100a1fe0 0x7f9910096130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9904009fb0 tx=0x7f9904017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f98f6ffd700 1 -- 172.21.15.121:0/1091403019 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990400b950 con 0x7f99100a1fe0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f98f6ffd700 1 -- 172.21.15.121:0/1091403019 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990400bad0 con 0x7f99100a1fe0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991013ccc0 con 0x7f99100a1fe0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f98f6ffd700 1 -- 172.21.15.121:0/1091403019 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990401a680 con 0x7f99100a1fe0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.240+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99100b6710 con 0x7f99100a1fe0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.241+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9910071660 con 0x7f99100a1fe0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.242+0000 7f98f6ffd700 1 -- 172.21.15.121:0/1091403019 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f990401aa30 con 0x7f99100a1fe0 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.242+0000 7f98f6ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.243+0000 7f98f6ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.243+0000 7f98f6ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.243+0000 7f98f6ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.243+0000 7f98f6ffd700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f98fc098780 0x7f98fc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.243+0000 7f990e7fc700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f98fc098780 0x7f98fc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.243+0000 7f98f6ffd700 1 -- 172.21.15.121:0/1091403019 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f99040c25b0 con 0x7f99100a1fe0 2023-09-29T23:11:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.243+0000 7f990e7fc700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f98fc098780 0x7f98fc09ac30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9910143680 tx=0x7f9900007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:18.246 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.246+0000 7f98f6ffd700 1 -- 172.21.15.121:0/1091403019 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f990408fc60 con 0x7f99100a1fe0 2023-09-29T23:11:18.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.468+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f99100b7930 con 0x7f99100a1fe0 2023-09-29T23:11:18.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.468+0000 7f98f6ffd700 1 -- 172.21.15.121:0/1091403019 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9904093280 con 0x7f99100a1fe0 2023-09-29T23:11:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:18.468 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 10 mgrc shutdown 2023-09-29T23:11:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f98fc098780 msgr2=0x7f98fc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f98fc098780 0x7f98fc09ac30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9910143680 tx=0x7f9900007ec0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100a1fe0 msgr2=0x7f9910096130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:18.470 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100a1fe0 0x7f9910096130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9904009fb0 tx=0x7f9904017000 comp rx=0 tx=0).stop 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 shutdown_connections 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f98fc098780 0x7f98fc09ac30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99101402b0 0x7f991013c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99100a1fe0 0x7f9910096130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 --2- 172.21.15.121:0/1091403019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99100b7560 0x7f991013bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 >> 172.21.15.121:0/1091403019 conn(0x7f9910068350 msgr2=0x7f991013f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 shutdown_connections 2023-09-29T23:11:18.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:18.470+0000 7f991532c700 1 -- 172.21.15.121:0/1091403019 wait complete. 2023-09-29T23:11:21.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:21.612 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.612+0000 7fce79943700 1 Processor -- start 2023-09-29T23:11:21.612 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.612+0000 7fce79943700 1 -- start start 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce79943700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce740a1c00 0x7fce741345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce79943700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce740a3e00 0x7fce7406c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce79943700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce740a1fd0 con 0x7fce7406bdc0 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce79943700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce740a4600 con 0x7fce740a1c00 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce79943700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce740a4340 con 0x7fce740a3e00 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce737fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce740a3e00 0x7fce7406c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce737fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce740a3e00 0x7fce7406c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34642/0 (socket says 172.21.15.121:34642) 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce737fe700 1 -- 172.21.15.121:0/3646541240 learned_addr learned my addr 172.21.15.121:0/3646541240 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce72ffd700 1 --2- 172.21.15.121:0/3646541240 >> v2:172.21.15.123:3300/0 conn(0x7fce740a1c00 0x7fce741345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce737fe700 1 -- 172.21.15.121:0/3646541240 >> v2:172.21.15.123:3300/0 conn(0x7fce740a1c00 msgr2=0x7fce741345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce737fe700 1 --2- 172.21.15.121:0/3646541240 >> v2:172.21.15.123:3300/0 conn(0x7fce740a1c00 0x7fce741345d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce737fe700 1 -- 172.21.15.121:0/3646541240 >> v1:172.21.15.123:6789/0 conn(0x7fce7406bdc0 legacy=0x7fce740a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.613+0000 7fce737fe700 1 -- 172.21.15.121:0/3646541240 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce740a4cd0 con 0x7fce740a3e00 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce737fe700 1 --2- 172.21.15.121:0/3646541240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce740a3e00 0x7fce7406c1c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009f20 tx=0x7fce6c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfbb1676a8963542 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce71ffb700 1 -- 172.21.15.121:0/3646541240 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c00e070 con 0x7fce740a3e00 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce71ffb700 1 -- 172.21.15.121:0/3646541240 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c00ba40 con 0x7fce740a3e00 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce71ffb700 1 -- 172.21.15.121:0/3646541240 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c0092e0 con 0x7fce740a3e00 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 -- 172.21.15.121:0/3646541240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce740a3e00 msgr2=0x7fce7406c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 --2- 172.21.15.121:0/3646541240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce740a3e00 0x7fce7406c1c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009f20 tx=0x7fce6c009840 comp rx=0 tx=0).stop 2023-09-29T23:11:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 -- 172.21.15.121:0/3646541240 shutdown_connections 2023-09-29T23:11:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 --2- 172.21.15.121:0/3646541240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce740a3e00 0x7fce7406c1c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 --2- 172.21.15.121:0/3646541240 >> v2:172.21.15.123:3300/0 conn(0x7fce740a1c00 0x7fce741345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 -- 172.21.15.121:0/3646541240 >> 172.21.15.121:0/3646541240 conn(0x7fce74067440 msgr2=0x7fce74067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 -- 172.21.15.121:0/3646541240 shutdown_connections 2023-09-29T23:11:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.614+0000 7fce79943700 1 -- 172.21.15.121:0/3646541240 wait complete. 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 Processor -- start 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 -- start start 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce7406bdc0 0x7fce7409af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce740a1c00 0x7fce74145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce740a3e00 0x7fce741464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74146a20 con 0x7fce740a3e00 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7409a570 con 0x7fce740a1c00 2023-09-29T23:11:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce79943700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7406e280 con 0x7fce7406bdc0 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce72ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce7406bdc0 0x7fce7409af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce740a3e00 0x7fce741464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce72ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce7406bdc0 0x7fce7409af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34656/0 (socket says 172.21.15.121:34656) 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.616+0000 7fce72ffd700 1 -- 172.21.15.121:0/458694248 learned_addr learned my addr 172.21.15.121:0/458694248 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce727fc700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce740a1c00 0x7fce74145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce72ffd700 1 -- 172.21.15.121:0/458694248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce740a1c00 msgr2=0x7fce74145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce72ffd700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce740a1c00 0x7fce74145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce72ffd700 1 -- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce740a3e00 msgr2=0x7fce741464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce72ffd700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce740a3e00 0x7fce741464e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce72ffd700 1 -- 172.21.15.121:0/458694248 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c009ae0 con 0x7fce7406bdc0 2023-09-29T23:11:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce737fe700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce740a3e00 0x7fce741464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce72ffd700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce7406bdc0 0x7fce7409af50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce6800d8f0 tx=0x7fce68008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce5affd700 1 -- 172.21.15.121:0/458694248 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce680125f0 con 0x7fce7406bdc0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce5affd700 1 -- 172.21.15.121:0/458694248 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce68012750 con 0x7fce7406bdc0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7406e540 con 0x7fce7406bdc0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce5affd700 1 -- 172.21.15.121:0/458694248 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68012af0 con 0x7fce7406bdc0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.617+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce74099e20 con 0x7fce7406bdc0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.618+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce74071660 con 0x7fce7406bdc0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce5affd700 1 -- 172.21.15.121:0/458694248 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fce6801e460 con 0x7fce7406bdc0 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce5affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce5affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce5affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce5affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce5affd700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fce60098800 0x7fce6009acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce727fc700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fce60098800 0x7fce6009acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.619+0000 7fce5affd700 1 -- 172.21.15.121:0/458694248 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fce680c0b70 con 0x7fce7406bdc0 2023-09-29T23:11:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.620+0000 7fce727fc700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fce60098800 0x7fce6009acb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fce7406e920 tx=0x7fce64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.624+0000 7fce5affd700 1 -- 172.21.15.121:0/458694248 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce6808e0f0 con 0x7fce7406bdc0 2023-09-29T23:11:21.843 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.843+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce740a3190 con 0x7fce7406bdc0 2023-09-29T23:11:21.843 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.843+0000 7fce5affd700 1 -- 172.21.15.121:0/458694248 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fce68091710 con 0x7fce7406bdc0 2023-09-29T23:11:21.843 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:21.844 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:21.845 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 10 mgrc shutdown 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fce60098800 msgr2=0x7fce6009acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fce60098800 0x7fce6009acb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fce7406e920 tx=0x7fce64009040 comp rx=0 tx=0).stop 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce7406bdc0 msgr2=0x7fce7409af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce7406bdc0 0x7fce7409af50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce6800d8f0 tx=0x7fce68008040 comp rx=0 tx=0).stop 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 shutdown_connections 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fce60098800 0x7fce6009acb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce740a3e00 0x7fce741464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fce7406bdc0 0x7fce7409af50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 --2- 172.21.15.121:0/458694248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce740a1c00 0x7fce74145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:21.846 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 >> 172.21.15.121:0/458694248 conn(0x7fce74067440 msgr2=0x7fce74070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:21.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 shutdown_connections 2023-09-29T23:11:21.847 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:21.845+0000 7fce79943700 1 -- 172.21.15.121:0/458694248 wait complete. 2023-09-29T23:11:24.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:24.987 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.987+0000 7fe23e868700 1 Processor -- start 2023-09-29T23:11:24.987 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.987+0000 7fe23e868700 1 -- start start 2023-09-29T23:11:24.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe23e868700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe23806ac70 0x7fe23806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:24.989 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe23e868700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe23806f8a0 0x7fe23814ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe23e868700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe238141530 con 0x7fe2380a53e0 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe23e868700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23806e110 con 0x7fe23806ac70 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe23e868700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23806e3d0 con 0x7fe23806f8a0 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe23806ac70 0x7fe23806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe23806ac70 0x7fe23806e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40842/0 (socket says 172.21.15.121:40842) 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 -- 172.21.15.121:0/1384446170 learned_addr learned my addr 172.21.15.121:0/1384446170 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe23ce05700 1 --2- 172.21.15.121:0/1384446170 >> v2:172.21.15.123:3300/0 conn(0x7fe23806f8a0 0x7fe23814ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 -- 172.21.15.121:0/1384446170 >> v2:172.21.15.123:3300/0 conn(0x7fe23806f8a0 msgr2=0x7fe23814ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 --2- 172.21.15.121:0/1384446170 >> v2:172.21.15.123:3300/0 conn(0x7fe23806f8a0 0x7fe23814ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 -- 172.21.15.121:0/1384446170 >> v1:172.21.15.123:6789/0 conn(0x7fe2380a53e0 legacy=0x7fe238140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 -- 172.21.15.121:0/1384446170 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe238141270 con 0x7fe23806ac70 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe237fff700 1 --2- 172.21.15.121:0/1384446170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe23806e9b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe23806bde0 tx=0x7fe22c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09e5de117d40865 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe236ffd700 1 -- 172.21.15.121:0/1384446170 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c00b860 con 0x7fe23806ac70 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe236ffd700 1 -- 172.21.15.121:0/1384446170 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c00b9c0 con 0x7fe23806ac70 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.988+0000 7fe236ffd700 1 -- 172.21.15.121:0/1384446170 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c00bd60 con 0x7fe23806ac70 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 -- 172.21.15.121:0/1384446170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 msgr2=0x7fe23806e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 --2- 172.21.15.121:0/1384446170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe23806e9b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe23806bde0 tx=0x7fe22c009250 comp rx=0 tx=0).stop 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 -- 172.21.15.121:0/1384446170 shutdown_connections 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 --2- 172.21.15.121:0/1384446170 >> v2:172.21.15.123:3300/0 conn(0x7fe23806f8a0 0x7fe23814ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 --2- 172.21.15.121:0/1384446170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe23806e9b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 -- 172.21.15.121:0/1384446170 >> 172.21.15.121:0/1384446170 conn(0x7fe238067440 msgr2=0x7fe238067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 -- 172.21.15.121:0/1384446170 shutdown_connections 2023-09-29T23:11:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.989+0000 7fe23e868700 1 -- 172.21.15.121:0/1384446170 wait complete. 2023-09-29T23:11:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.990+0000 7fe23e868700 1 Processor -- start 2023-09-29T23:11:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.990+0000 7fe23e868700 1 -- start start 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe2380a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe23806f8a0 0x7fe23809ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2380a53e0 0x7fe23813e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23809a0e0 con 0x7fe23806ac70 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380a42b0 con 0x7fe23806f8a0 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23809ba40 con 0x7fe2380a53e0 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe2380a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe2380a0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40858/0 (socket says 172.21.15.121:40858) 2023-09-29T23:11:24.995 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23ce05700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2380a53e0 0x7fe23813e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 -- 172.21.15.121:0/2314372720 learned_addr learned my addr 172.21.15.121:0/2314372720 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe2377fe700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe23806f8a0 0x7fe23809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 -- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2380a53e0 msgr2=0x7fe23813e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2380a53e0 0x7fe23813e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 -- 172.21.15.121:0/2314372720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe23806f8a0 msgr2=0x7fe23809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe23806f8a0 0x7fe23809ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 -- 172.21.15.121:0/2314372720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c0094f0 con 0x7fe23806ac70 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23ce05700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2380a53e0 0x7fe23813e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe237fff700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe2380a0fb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe2380a7a20 tx=0x7fe22c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe21ffff700 1 -- 172.21.15.121:0/2314372720 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c024440 con 0x7fe23806ac70 2023-09-29T23:11:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe21ffff700 1 -- 172.21.15.121:0/2314372720 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c0245c0 con 0x7fe23806ac70 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe21ffff700 1 -- 172.21.15.121:0/2314372720 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c024980 con 0x7fe23806ac70 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2380a3ff0 con 0x7fe23806ac70 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.991+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23809b1c0 con 0x7fe23806ac70 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.992+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe238071660 con 0x7fe23806ac70 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.993+0000 7fe21ffff700 1 -- 172.21.15.121:0/2314372720 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe22c01a420 con 0x7fe23806ac70 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.993+0000 7fe21ffff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.994+0000 7fe21ffff700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.994+0000 7fe21ffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.994+0000 7fe21ffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.994+0000 7fe21ffff700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe224098800 0x7fe22409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.994+0000 7fe2377fe700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe224098800 0x7fe22409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:24.997 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.994+0000 7fe21ffff700 1 -- 172.21.15.121:0/2314372720 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7fe22c0c21f0 con 0x7fe23806ac70 2023-09-29T23:11:24.998 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.994+0000 7fe2377fe700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe224098800 0x7fe22409acb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe2381419d0 tx=0x7fe228009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:24.998 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:24.998+0000 7fe21ffff700 1 -- 172.21.15.121:0/2314372720 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe22c08f800 con 0x7fe23806ac70 2023-09-29T23:11:25.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.218+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe23806eef0 con 0x7fe23806ac70 2023-09-29T23:11:25.218 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.219+0000 7fe21ffff700 1 -- 172.21.15.121:0/2314372720 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe22c092e20 con 0x7fe23806ac70 2023-09-29T23:11:25.219 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:25.219 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 10 mgrc shutdown 2023-09-29T23:11:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe224098800 msgr2=0x7fe22409acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe224098800 0x7fe22409acb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe2381419d0 tx=0x7fe228009040 comp rx=0 tx=0).stop 2023-09-29T23:11:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 msgr2=0x7fe2380a0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe2380a0fb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe2380a7a20 tx=0x7fe22c017000 comp rx=0 tx=0).stop 2023-09-29T23:11:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 shutdown_connections 2023-09-29T23:11:25.221 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7fe224098800 0x7fe22409acb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:25.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2380a53e0 0x7fe23813e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:25.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23806ac70 0x7fe2380a0fb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:25.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 --2- 172.21.15.121:0/2314372720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe23806f8a0 0x7fe23809ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:25.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 >> 172.21.15.121:0/2314372720 conn(0x7fe238067440 msgr2=0x7fe2380b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:25.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 shutdown_connections 2023-09-29T23:11:25.222 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:25.221+0000 7fe23e868700 1 -- 172.21.15.121:0/2314372720 wait complete. 2023-09-29T23:11:28.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:28.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.361+0000 7f4dc016d700 1 Processor -- start 2023-09-29T23:11:28.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.361+0000 7f4dc016d700 1 -- start start 2023-09-29T23:11:28.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dc016d700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4db8141440 0x7f4db80a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dc016d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4db80b3cf0 0x7f4db814b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dc016d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db8141810 con 0x7f4db809b770 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dc016d700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db809a7f0 con 0x7f4db80b3cf0 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dc016d700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80a4df0 con 0x7f4db8141440 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbe70a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4db80b3cf0 0x7f4db814b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbd708700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4db809b770 0x7f4db80630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50522/0 (socket says 172.21.15.121:50522) 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbd708700 1 -- 172.21.15.121:0/2918593316 learned_addr learned my addr 172.21.15.121:0/2918593316 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbdf09700 1 --2- 172.21.15.121:0/2918593316 >> v2:172.21.15.123:3300/0 conn(0x7f4db8141440 0x7f4db80a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbe70a700 1 -- 172.21.15.121:0/2918593316 >> v2:172.21.15.123:3300/0 conn(0x7f4db8141440 msgr2=0x7f4db80a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:28.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbe70a700 1 --2- 172.21.15.121:0/2918593316 >> v2:172.21.15.123:3300/0 conn(0x7f4db8141440 0x7f4db80a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbe70a700 1 -- 172.21.15.121:0/2918593316 >> v1:172.21.15.121:6789/0 conn(0x7f4db809b770 legacy=0x7f4db80630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbe70a700 1 -- 172.21.15.121:0/2918593316 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80a50b0 con 0x7f4db80b3cf0 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbcf07700 1 -- 172.21.15.121:0/2918593316 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321740953 0 0) 0x7f4db8141810 con 0x7f4db809b770 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbe70a700 1 --2- 172.21.15.121:0/2918593316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 0x7f4db814b190 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4db806bea0 tx=0x7f4db0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901f0f478dce608b server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.362+0000 7f4dbcf07700 1 -- 172.21.15.121:0/2918593316 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db000e070 con 0x7f4db80b3cf0 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dbcf07700 1 -- 172.21.15.121:0/2918593316 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db000bc30 con 0x7f4db80b3cf0 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dbcf07700 1 -- 172.21.15.121:0/2918593316 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0012430 con 0x7f4db80b3cf0 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 -- 172.21.15.121:0/2918593316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 msgr2=0x7f4db814b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 --2- 172.21.15.121:0/2918593316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 0x7f4db814b190 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4db806bea0 tx=0x7f4db0009780 comp rx=0 tx=0).stop 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 -- 172.21.15.121:0/2918593316 shutdown_connections 2023-09-29T23:11:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 --2- 172.21.15.121:0/2918593316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 0x7f4db814b190 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 --2- 172.21.15.121:0/2918593316 >> v2:172.21.15.123:3300/0 conn(0x7f4db8141440 0x7f4db80a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 -- 172.21.15.121:0/2918593316 >> 172.21.15.121:0/2918593316 conn(0x7f4db8069e90 msgr2=0x7f4db80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 -- 172.21.15.121:0/2918593316 shutdown_connections 2023-09-29T23:11:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.363+0000 7f4dc016d700 1 -- 172.21.15.121:0/2918593316 wait complete. 2023-09-29T23:11:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.364+0000 7f4dc016d700 1 Processor -- start 2023-09-29T23:11:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.364+0000 7f4dc016d700 1 -- start start 2023-09-29T23:11:28.368 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dc016d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4db809b770 0x7f4db809df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dc016d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 0x7f4db80b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dc016d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4db8141440 0x7f4db814f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dc016d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db809dae0 con 0x7f4db80b3cf0 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dc016d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80a2de0 con 0x7f4db8141440 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dc016d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8153bb0 con 0x7f4db809b770 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4db809b770 0x7f4db809df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbd708700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 0x7f4db80b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4db809b770 0x7f4db809df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36764/0 (socket says 172.21.15.121:36764) 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 -- 172.21.15.121:0/3442874081 learned_addr learned my addr 172.21.15.121:0/3442874081 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbe70a700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4db8141440 0x7f4db814f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 -- 172.21.15.121:0/3442874081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4db8141440 msgr2=0x7f4db814f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4db8141440 0x7f4db814f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 -- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 msgr2=0x7f4db80b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 0x7f4db80b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 -- 172.21.15.121:0/3442874081 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0009a20 con 0x7f4db809b770 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dbdf09700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4db809b770 0x7f4db809df50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4da800c880 tx=0x7f4da8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dadffb700 1 -- 172.21.15.121:0/3442874081 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8011600 con 0x7f4db809b770 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dadffb700 1 -- 172.21.15.121:0/3442874081 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da8011760 con 0x7f4db809b770 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db8153ea0 con 0x7f4db809b770 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.365+0000 7f4dadffb700 1 -- 172.21.15.121:0/3442874081 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8011b00 con 0x7f4db809b770 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.366+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db80b2ac0 con 0x7f4db809b770 2023-09-29T23:11:28.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.367+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db809dda0 con 0x7f4db809b770 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.367+0000 7f4dadffb700 1 -- 172.21.15.121:0/3442874081 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f4da8018360 con 0x7f4db809b770 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.367+0000 7f4dadffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.368+0000 7f4dadffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.368+0000 7f4dadffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.368+0000 7f4dadffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.368+0000 7f4dadffb700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4d9c0987b0 0x7f4d9c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.368+0000 7f4dbd708700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4d9c0987b0 0x7f4d9c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.368+0000 7f4dadffb700 1 -- 172.21.15.121:0/3442874081 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f4da80c0260 con 0x7f4db809b770 2023-09-29T23:11:28.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.368+0000 7f4dbd708700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4d9c0987b0 0x7f4d9c09ac60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4db80a31e0 tx=0x7f4db4007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:28.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.372+0000 7f4dadffb700 1 -- 172.21.15.121:0/3442874081 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4da80913a0 con 0x7f4db809b770 2023-09-29T23:11:28.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.590+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4db806ab60 con 0x7f4db809b770 2023-09-29T23:11:28.590 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.590+0000 7f4dadffb700 1 -- 172.21.15.121:0/3442874081 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4da8015100 con 0x7f4db809b770 2023-09-29T23:11:28.590 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:28.591 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:28.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 10 mgrc shutdown 2023-09-29T23:11:28.592 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4d9c0987b0 msgr2=0x7f4d9c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4d9c0987b0 0x7f4d9c09ac60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4db80a31e0 tx=0x7f4db4007d30 comp rx=0 tx=0).stop 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4db809b770 msgr2=0x7f4db809df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4db809b770 0x7f4db809df50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4da800c880 tx=0x7f4da8008040 comp rx=0 tx=0).stop 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 shutdown_connections 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f4d9c0987b0 0x7f4d9c09ac60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4db8141440 0x7f4db814f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4db809b770 0x7f4db809df50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.592+0000 7f4dc016d700 1 --2- 172.21.15.121:0/3442874081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4db80b3cf0 0x7f4db80b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.593+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 >> 172.21.15.121:0/3442874081 conn(0x7f4db8069e90 msgr2=0x7f4db80a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:28.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.593+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 shutdown_connections 2023-09-29T23:11:28.594 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:28.593+0000 7f4dc016d700 1 -- 172.21.15.121:0/3442874081 wait complete. 2023-09-29T23:11:31.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-29T23:11:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.733+0000 7f9426cdc700 1 Processor -- start 2023-09-29T23:11:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.733+0000 7f9426cdc700 1 -- start start 2023-09-29T23:11:31.736 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f9426cdc700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f942006f8a0 0x7f942014be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f9426cdc700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9420141530 con 0x7f94200a53e0 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f9426cdc700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942006e110 con 0x7f942006ac70 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f9426cdc700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f942006e3d0 con 0x7f942006f8a0 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f941ffff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f94200a53e0 0x7f9420140b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50530/0 (socket says 172.21.15.121:50530) 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f941ffff700 1 -- 172.21.15.121:0/2510070954 learned_addr learned my addr 172.21.15.121:0/2510070954 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900917456 0 0) 0x7f9420141530 con 0x7f94200a53e0 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9410003570 con 0x7f94200a53e0 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.734+0000 7f9425279700 1 --2- 172.21.15.121:0/2510070954 >> v2:172.21.15.123:3300/0 conn(0x7f942006f8a0 0x7f942014be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 414735155 0 0) 0x7f9410003570 con 0x7f94200a53e0 2023-09-29T23:11:31.737 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 >> v2:172.21.15.123:3300/0 conn(0x7f942006f8a0 msgr2=0x7f942014be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 --2- 172.21.15.121:0/2510070954 >> v2:172.21.15.123:3300/0 conn(0x7f942006f8a0 0x7f942014be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 >> v1:172.21.15.123:6789/0 conn(0x7f942006ac70 legacy=0x7f942006e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420141270 con 0x7f94200a53e0 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 41485407 0 0) 0x7f940c001470 con 0x7f94200a53e0 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 >> v1:172.21.15.121:6789/0 conn(0x7f94200a53e0 legacy=0x7f9420140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9410003a60 0x7f9410003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9410004810 0x7f9410004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94100051c0 0x7f941000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942006e3d0 con 0x7f94100051c0 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941000c0d0 con 0x7f9410003a60 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941000c290 con 0x7f9410004810 2023-09-29T23:11:31.738 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941ffff700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9410004810 0x7f9410004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94100051c0 0x7f941000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9424a78700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9410003a60 0x7f9410003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 -- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9410004810 msgr2=0x7f9410004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9410004810 0x7f9410004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 -- 172.21.15.121:0/2510070954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9410003a60 msgr2=0x7f9410003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9410003a60 0x7f9410003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 -- 172.21.15.121:0/2510070954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410003570 con 0x7f94100051c0 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941ffff700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9410004810 0x7f9410004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 -- 172.21.15.121:0/2510070954 reap_dead start 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f9425279700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94100051c0 0x7f941000ba60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f941800e210 tx=0x7f941800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be746e38a219898 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.735+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941800eac0 con 0x7f94100051c0 2023-09-29T23:11:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941800f040 con 0x7f94100051c0 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f941f7fe700 1 -- 172.21.15.121:0/2510070954 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941800ee70 con 0x7f94100051c0 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f9426cdc700 1 -- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94100051c0 msgr2=0x7f941000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f9426cdc700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94100051c0 0x7f941000ba60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f941800e210 tx=0x7f941800b040 comp rx=0 tx=0).stop 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f9426cdc700 1 -- 172.21.15.121:0/2510070954 shutdown_connections 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f9426cdc700 1 --2- 172.21.15.121:0/2510070954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94100051c0 0x7f941000ba60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f9426cdc700 1 -- 172.21.15.121:0/2510070954 >> 172.21.15.121:0/2510070954 conn(0x7f9420067440 msgr2=0x7f9420067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f9426cdc700 1 -- 172.21.15.121:0/2510070954 shutdown_connections 2023-09-29T23:11:31.740 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.736+0000 7f9426cdc700 1 -- 172.21.15.121:0/2510070954 wait complete. 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.737+0000 7f9426cdc700 1 Processor -- start 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9426cdc700 1 -- start start 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9426cdc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94100051c0 0x7f942009a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9426cdc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f942013d060 0x7f9420150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9426cdc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9420150c40 0x7f94201431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9426cdc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94200a3ca0 con 0x7f9420150c40 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9426cdc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9420151650 con 0x7f942013d060 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9426cdc700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942013d470 con 0x7f94100051c0 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9424a78700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94100051c0 0x7f942009a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9424a78700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94100051c0 0x7f942009a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36782/0 (socket says 172.21.15.121:36782) 2023-09-29T23:11:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9425279700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9420150c40 0x7f94201431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.738+0000 7f9424a78700 1 -- 172.21.15.121:0/3999260014 learned_addr learned my addr 172.21.15.121:0/3999260014 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9424a78700 1 -- 172.21.15.121:0/3999260014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f942013d060 msgr2=0x7f9420150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9424a78700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f942013d060 0x7f9420150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9424a78700 1 -- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9420150c40 msgr2=0x7f94201431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9424a78700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9420150c40 0x7f94201431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9424a78700 1 -- 172.21.15.121:0/3999260014 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420151390 con 0x7f94100051c0 2023-09-29T23:11:31.745 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9424a78700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94100051c0 0x7f942009a070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9414002980 tx=0x7f9414007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f941cff9700 1 -- 172.21.15.121:0/3999260014 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9414002e10 con 0x7f94100051c0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f941cff9700 1 -- 172.21.15.121:0/3999260014 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941400f8d0 con 0x7f94100051c0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9420151130 con 0x7f94100051c0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f941cff9700 1 -- 172.21.15.121:0/3999260014 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941400fc70 con 0x7f94100051c0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.739+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9420143700 con 0x7f94100051c0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.740+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9420071660 con 0x7f94100051c0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.741+0000 7f941cff9700 1 -- 172.21.15.121:0/3999260014 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f94140133f0 con 0x7f94100051c0 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.741+0000 7f941cff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.741+0000 7f941cff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.741+0000 7f941cff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.741+0000 7f941cff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.741+0000 7f941cff9700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9404098780 0x7f940409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.742+0000 7f941ffff700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9404098780 0x7f940409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.742+0000 7f941cff9700 1 -- 172.21.15.121:0/3999260014 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f94140c01e0 con 0x7f94100051c0 2023-09-29T23:11:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.742+0000 7f941ffff700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9404098780 0x7f940409ac30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f940c007fd0 tx=0x7f942006e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.746+0000 7f941cff9700 1 -- 172.21.15.121:0/3999260014 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f941408d800 con 0x7f94100051c0 2023-09-29T23:11:31.963 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.964+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9420141f90 con 0x7f94100051c0 2023-09-29T23:11:31.964 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.964+0000 7f941cff9700 1 -- 172.21.15.121:0/3999260014 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9414090e20 con 0x7f94100051c0 2023-09-29T23:11:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:31.964 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-29T23:11:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 10 mgrc shutdown 2023-09-29T23:11:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9404098780 msgr2=0x7f940409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:31.966 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9404098780 0x7f940409ac30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f940c007fd0 tx=0x7f942006e000 comp rx=0 tx=0).stop 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94100051c0 msgr2=0x7f942009a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94100051c0 0x7f942009a070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9414002980 tx=0x7f9414007d30 comp rx=0 tx=0).stop 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 shutdown_connections 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f9404098780 0x7f940409ac30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9420150c40 0x7f94201431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94100051c0 0x7f942009a070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 --2- 172.21.15.121:0/3999260014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f942013d060 0x7f9420150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 >> 172.21.15.121:0/3999260014 conn(0x7f9420067440 msgr2=0x7f942013fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 shutdown_connections 2023-09-29T23:11:31.967 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:31.966+0000 7f9426cdc700 1 -- 172.21.15.121:0/3999260014 wait complete. 2023-09-29T23:11:31.978 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-29T23:11:31.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-29T23:11:32.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.096+0000 7ff5bd435700 1 Processor -- start 2023-09-29T23:11:32.096 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.096+0000 7ff5bd435700 1 -- start start 2023-09-29T23:11:32.098 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5bd435700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff5b80a2110 0x7ff5b814c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5bd435700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80a2b40 con 0x7ff5b8068240 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5bd435700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80a2880 con 0x7ff5b80a91a0 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5bd435700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b8068610 con 0x7ff5b80a2110 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b77fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff5b80a2110 0x7ff5b814c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b6ffd700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff5b8068240 0x7ff5b813fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:50546/0 (socket says 172.21.15.121:50546) 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b77fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff5b80a2110 0x7ff5b814c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52588/0 (socket says 172.21.15.121:52588) 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b6ffd700 1 -- 172.21.15.121:0/3793412690 learned_addr learned my addr 172.21.15.121:0/3793412690 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b77fe700 1 -- 172.21.15.121:0/3793412690 >> v1:172.21.15.123:6789/0 conn(0x7ff5b80a91a0 legacy=0x7ff5b80a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b77fe700 1 -- 172.21.15.121:0/3793412690 >> v1:172.21.15.121:6789/0 conn(0x7ff5b8068240 legacy=0x7ff5b813fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b5ffb700 1 -- 172.21.15.121:0/3793412690 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038427397 0 0) 0x7ff5b80a2b40 con 0x7ff5b8068240 2023-09-29T23:11:32.099 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.097+0000 7ff5b77fe700 1 -- 172.21.15.121:0/3793412690 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b80a8c70 con 0x7ff5b80a2110 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5b77fe700 1 --2- 172.21.15.121:0/3793412690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a2110 0x7ff5b814c0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009e60 tx=0x7ff5b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab146391e2db3ae1 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5b5ffb700 1 -- 172.21.15.121:0/3793412690 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b000e070 con 0x7ff5b80a2110 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5b5ffb700 1 -- 172.21.15.121:0/3793412690 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b000bc30 con 0x7ff5b80a2110 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5b5ffb700 1 -- 172.21.15.121:0/3793412690 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0012430 con 0x7ff5b80a2110 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5bd435700 1 -- 172.21.15.121:0/3793412690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a2110 msgr2=0x7ff5b814c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5bd435700 1 --2- 172.21.15.121:0/3793412690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a2110 0x7ff5b814c0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009e60 tx=0x7ff5b0009780 comp rx=0 tx=0).stop 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5bd435700 1 -- 172.21.15.121:0/3793412690 shutdown_connections 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5bd435700 1 --2- 172.21.15.121:0/3793412690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a2110 0x7ff5b814c0d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5bd435700 1 -- 172.21.15.121:0/3793412690 >> 172.21.15.121:0/3793412690 conn(0x7ff5b806f6a0 msgr2=0x7ff5b8067970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5bd435700 1 -- 172.21.15.121:0/3793412690 shutdown_connections 2023-09-29T23:11:32.100 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.098+0000 7ff5bd435700 1 -- 172.21.15.121:0/3793412690 wait complete. 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.099+0000 7ff5bd435700 1 Processor -- start 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.099+0000 7ff5bd435700 1 -- start start 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5bd435700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5b8068240 0x7ff5b813eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5bd435700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5b80a2110 0x7ff5b81393a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5bd435700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a91a0 0x7ff5b8142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5bd435700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b8153670 con 0x7ff5b80a91a0 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5bd435700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b81544f0 con 0x7ff5b80a2110 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5bd435700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b8153860 con 0x7ff5b8068240 2023-09-29T23:11:32.103 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b6ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5b8068240 0x7ff5b813eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a91a0 0x7ff5b8142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a91a0 0x7ff5b8142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52602/0 (socket says 172.21.15.121:52602) 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b6ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5b8068240 0x7ff5b813eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36794/0 (socket says 172.21.15.121:36794) 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 -- 172.21.15.121:0/1099721799 learned_addr learned my addr 172.21.15.121:0/1099721799 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b67fc700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5b80a2110 0x7ff5b81393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 -- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5b8068240 msgr2=0x7ff5b813eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5b8068240 0x7ff5b813eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 -- 172.21.15.121:0/1099721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5b80a2110 msgr2=0x7ff5b81393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5b80a2110 0x7ff5b81393a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b77fe700 1 -- 172.21.15.121:0/1099721799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b0009a20 con 0x7ff5b80a91a0 2023-09-29T23:11:32.104 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.100+0000 7ff5b6ffd700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5b8068240 0x7ff5b813eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.101+0000 7ff5b77fe700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a91a0 0x7ff5b8142fd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0013000 tx=0x7ff5b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.101+0000 7ff5aeffd700 1 -- 172.21.15.121:0/1099721799 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0012e60 con 0x7ff5b80a91a0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.101+0000 7ff5aeffd700 1 -- 172.21.15.121:0/1099721799 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b001b630 con 0x7ff5b80a91a0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.101+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b8153b20 con 0x7ff5b80a91a0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.101+0000 7ff5aeffd700 1 -- 172.21.15.121:0/1099721799 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b001b9d0 con 0x7ff5b80a91a0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.101+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b8142000 con 0x7ff5b80a91a0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.102+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b806a1b0 con 0x7ff5b80a91a0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5aeffd700 1 -- 172.21.15.121:0/1099721799 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff5b0003020 con 0x7ff5b80a91a0 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5aeffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5aeffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5aeffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5aeffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:32.105 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5aeffd700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff5980987b0 0x7ff59809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.106 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5aeffd700 1 -- 172.21.15.121:0/1099721799 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7ff5b00c2350 con 0x7ff5b80a91a0 2023-09-29T23:11:32.106 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.103+0000 7ff5b6ffd700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff5980987b0 0x7ff59809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.106 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.104+0000 7ff5b6ffd700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff5980987b0 0x7ff59809ac60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00016f0 tx=0x7ff5a0007f50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:32.106 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.107+0000 7ff5aeffd700 1 -- 172.21.15.121:0/1099721799 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5b008fa00 con 0x7ff5b80a91a0 2023-09-29T23:11:32.288 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.289+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b8069080 con 0x7ff5b80a91a0 2023-09-29T23:11:32.289 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.289+0000 7ff5aeffd700 1 -- 172.21.15.121:0/1099721799 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+12490 (secure 0 0 0) 0x7ff5b002a090 con 0x7ff5b80a91a0 2023-09-29T23:11:32.289 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-29T23:11:32.289 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":21,"fsid":"f139d264-5213-413d-9960-69a6d67478bb","created":"2023-09-29T23:05:56.090420+0000","modified":"2023-09-29T23:07:02.951788+0000","last_up_change":"2023-09-29T23:06:07.289163+0000","last_in_change":"2023-09-29T23:05:57.470851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-29T23:06:07.527057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-29T23:06:12.101557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-29T23:06:16.382138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-29T23:06:17.218163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"668222c3-dafb-41e3-9a19-feeeafa50f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":113368},{"type":"v1","addr":"172.21.15.121:6801","nonce":113368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":113368},{"type":"v1","addr":"172.21.15.121:6803","nonce":113368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":113368},{"type":"v1","addr":"172.21.15.121:6807","nonce":113368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":113368},{"type":"v1","addr":"172.21.15.121:6805","nonce":113368}]},"public_addr":"172.21.15.121:6801/113368","cluster_addr":"172.21.15.121:6803/113368","heartbeat_back_addr":"172.21.15.121:6807/113368","heartbeat_front_addr":"172.21.15.121:6805/113368","state":["exists","up"]},{"osd":1,"uuid":"eb400e3d-c1be-4433-99df-ccbdabfa1054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":115891},{"type":"v1","addr":"172.21.15.123:6817","nonce":115891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":115891},{"type":"v1","addr":"172.21.15.123:6819","nonce":115891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":115891},{"type":"v1","addr":"172.21.15.123:6823","nonce":115891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":115891},{"type":"v1","addr":"172.21.15.123:6821","nonce":115891}]},"public_addr":"172.21.15.123:6817/115891","cluster_addr":"172.21.15.123:6819/115891","heartbeat_back_addr":"172.21.15.123:6823/115891","heartbeat_front_addr":"172.21.15.123:6821/115891","state":["exists","up"]},{"osd":2,"uuid":"8304c4c7-39e6-4a99-bcf5-d39fb111241c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":115865},{"type":"v1","addr":"172.21.15.123:6801","nonce":115865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":115865},{"type":"v1","addr":"172.21.15.123:6803","nonce":115865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":115865},{"type":"v1","addr":"172.21.15.123:6807","nonce":115865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":115865},{"type":"v1","addr":"172.21.15.123:6805","nonce":115865}]},"public_addr":"172.21.15.123:6801/115865","cluster_addr":"172.21.15.123:6803/115865","heartbeat_back_addr":"172.21.15.123:6807/115865","heartbeat_front_addr":"172.21.15.123:6805/115865","state":["exists","up"]},{"osd":3,"uuid":"0614da14-c1b7-4d44-a6ab-9e49ab25ede0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":115866},{"type":"v1","addr":"172.21.15.123:6809","nonce":115866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":115866},{"type":"v1","addr":"172.21.15.123:6811","nonce":115866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":115866},{"type":"v1","addr":"172.21.15.123:6815","nonce":115866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":115866},{"type":"v1","addr":"172.21.15.123:6813","nonce":115866}]},"public_addr":"172.21.15.123:6809/115866","cluster_addr":"172.21.15.123:6811/115866","heartbeat_back_addr":"172.21.15.123:6815/115866","heartbeat_front_addr":"172.21.15.123:6813/115866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.533803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.589155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.610974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T23:06:02.651844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.5","mappings":[{"from":0,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":2}]},{"pgid":"4.21","mappings":[{"from":3,"to":2}]},{"pgid":"4.22","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T23:11:32.291 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 10 mgrc shutdown 2023-09-29T23:11:32.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff5980987b0 msgr2=0x7ff59809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff5980987b0 0x7ff59809ac60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00016f0 tx=0x7ff5a0007f50 comp rx=0 tx=0).stop 2023-09-29T23:11:32.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a91a0 msgr2=0x7ff5b8142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a91a0 0x7ff5b8142fd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0013000 tx=0x7ff5b0019000 comp rx=0 tx=0).stop 2023-09-29T23:11:32.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 shutdown_connections 2023-09-29T23:11:32.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7ff5980987b0 0x7ff59809ac60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.292 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5b80a91a0 0x7ff5b8142fd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5b8068240 0x7ff5b813eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 --2- 172.21.15.121:0/1099721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5b80a2110 0x7ff5b81393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.291+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 >> 172.21.15.121:0/1099721799 conn(0x7ff5b806f6a0 msgr2=0x7ff5b80a0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:32.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.292+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 shutdown_connections 2023-09-29T23:11:32.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.292+0000 7ff5bd435700 1 -- 172.21.15.121:0/1099721799 wait complete. 2023-09-29T23:11:32.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-29T23:11:32.420 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.421+0000 7f2c8781e700 1 Processor -- start 2023-09-29T23:11:32.421 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.421+0000 7f2c8781e700 1 -- start start 2023-09-29T23:11:32.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c8781e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006acb0 0x7f2c8009e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c8781e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c8006b080 0x7f2c80141d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c8781e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c801422c0 0x7f2c8013e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.423 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c8781e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c800b4990 con 0x7f2c8006b080 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c8781e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8013e3d0 con 0x7f2c801422c0 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c8781e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c801417a0 con 0x7f2c8006acb0 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c84db9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c8006b080 0x7f2c80141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006acb0 0x7f2c8009e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c84db9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c8006b080 0x7f2c80141d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52616/0 (socket says 172.21.15.121:52616) 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c84db9700 1 -- 172.21.15.121:0/4057227630 learned_addr learned my addr 172.21.15.121:0/4057227630 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006acb0 0x7f2c8009e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36800/0 (socket says 172.21.15.121:36800) 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c85dbb700 1 --2- 172.21.15.121:0/4057227630 >> v2:172.21.15.123:3300/0 conn(0x7f2c801422c0 0x7f2c8013e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 -- 172.21.15.121:0/4057227630 >> v2:172.21.15.123:3300/0 conn(0x7f2c801422c0 msgr2=0x7f2c8013e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 --2- 172.21.15.121:0/4057227630 >> v2:172.21.15.123:3300/0 conn(0x7f2c801422c0 0x7f2c8013e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 -- 172.21.15.121:0/4057227630 >> v2:172.21.15.121:3300/0 conn(0x7f2c8006b080 msgr2=0x7f2c80141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-29T23:11:32.424 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 --2- 172.21.15.121:0/4057227630 >> v2:172.21.15.121:3300/0 conn(0x7f2c8006b080 0x7f2c80141d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 -- 172.21.15.121:0/4057227630 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c801418e0 con 0x7f2c8006acb0 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c84db9700 1 --2- 172.21.15.121:0/4057227630 >> v2:172.21.15.121:3300/0 conn(0x7f2c8006b080 0x7f2c80141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c855ba700 1 --2- 172.21.15.121:0/4057227630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006acb0 0x7f2c8009e300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2c8006cf60 tx=0x7f2c7c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c806ba47435b088 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.422+0000 7f2c77fff700 1 -- 172.21.15.121:0/4057227630 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c005c10 con 0x7f2c8006acb0 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c77fff700 1 -- 172.21.15.121:0/4057227630 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c005d70 con 0x7f2c8006acb0 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c77fff700 1 -- 172.21.15.121:0/4057227630 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c006260 con 0x7f2c8006acb0 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 -- 172.21.15.121:0/4057227630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006acb0 msgr2=0x7f2c8009e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 --2- 172.21.15.121:0/4057227630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006acb0 0x7f2c8009e300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2c8006cf60 tx=0x7f2c7c009120 comp rx=0 tx=0).stop 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 -- 172.21.15.121:0/4057227630 shutdown_connections 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 --2- 172.21.15.121:0/4057227630 >> v2:172.21.15.123:3300/0 conn(0x7f2c801422c0 0x7f2c8013e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 --2- 172.21.15.121:0/4057227630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006acb0 0x7f2c8009e300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.425 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 --2- 172.21.15.121:0/4057227630 >> v2:172.21.15.121:3300/0 conn(0x7f2c8006b080 0x7f2c80141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 -- 172.21.15.121:0/4057227630 >> 172.21.15.121:0/4057227630 conn(0x7f2c8006d160 msgr2=0x7f2c8006d560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:32.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 -- 172.21.15.121:0/4057227630 shutdown_connections 2023-09-29T23:11:32.426 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.423+0000 7f2c8781e700 1 -- 172.21.15.121:0/4057227630 wait complete. 2023-09-29T23:11:32.428 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.424+0000 7f2c8781e700 1 Processor -- start 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.424+0000 7f2c8781e700 1 -- start start 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c8781e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c8006acb0 0x7f2c8009fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c8781e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006b080 0x7f2c80068550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c8781e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c801422c0 0x7f2c80068a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c8781e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c800a29f0 con 0x7f2c801422c0 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c8781e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c800a3300 con 0x7f2c8006acb0 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c8781e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c800a35a0 con 0x7f2c8006b080 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c84db9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006b080 0x7f2c80068550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c85dbb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c801422c0 0x7f2c80068a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.429 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c84db9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006b080 0x7f2c80068550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:36810/0 (socket says 172.21.15.121:36810) 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c84db9700 1 -- 172.21.15.121:0/3705020821 learned_addr learned my addr 172.21.15.121:0/3705020821 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c855ba700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c8006acb0 0x7f2c8009fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c85dbb700 1 -- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006b080 msgr2=0x7f2c80068550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c85dbb700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006b080 0x7f2c80068550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c85dbb700 1 -- 172.21.15.121:0/3705020821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c8006acb0 msgr2=0x7f2c8009fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c85dbb700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c8006acb0 0x7f2c8009fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c85dbb700 1 -- 172.21.15.121:0/3705020821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c700094f0 con 0x7f2c801422c0 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c84db9700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006b080 0x7f2c80068550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c85dbb700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c801422c0 0x7f2c80068a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2c78007e70 tx=0x7f2c780097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:32.430 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78018070 con 0x7f2c801422c0 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7800eb70 con 0x7f2c801422c0 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c0093c0 con 0x7f2c801422c0 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.425+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78014420 con 0x7f2c801422c0 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.426+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c800a4720 con 0x7f2c801422c0 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.427+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c800a19d0 con 0x7f2c801422c0 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.427+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2c78003020 con 0x7f2c801422c0 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.427+0000 7f2c757fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.428+0000 7f2c757fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.428+0000 7f2c757fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.428+0000 7f2c757fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] 2023-09-29T23:11:32.431 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.428+0000 7f2c757fa700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c64098780 0x7f2c6409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-29T23:11:32.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.428+0000 7f2c855ba700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c64098780 0x7f2c6409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-29T23:11:32.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.428+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4475+0+0 (secure 0 0 0) 0x7f2c780c04f0 con 0x7f2c801422c0 2023-09-29T23:11:32.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.428+0000 7f2c855ba700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c64098780 0x7f2c6409ac30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00a010 tx=0x7f2c7c016000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-29T23:11:32.432 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.432+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c78091630 con 0x7f2c801422c0 2023-09-29T23:11:32.615 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.615+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f2c800b5950 con 0x7f2c801422c0 2023-09-29T23:11:32.618 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.619+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2c78091630 con 0x7f2c801422c0 2023-09-29T23:11:32.619 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.620+0000 7f2c757fa700 1 -- 172.21.15.121:0/3705020821 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) v1 ==== 123+0+0 (secure 0 0 0) 0x7f2c780098f0 con 0x7f2c801422c0 2023-09-29T23:11:32.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 10 mgrc shutdown 2023-09-29T23:11:32.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c64098780 msgr2=0x7f2c6409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c64098780 0x7f2c6409ac30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00a010 tx=0x7f2c7c016000 comp rx=0 tx=0).stop 2023-09-29T23:11:32.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c801422c0 msgr2=0x7f2c80068a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-29T23:11:32.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c801422c0 0x7f2c80068a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2c78007e70 tx=0x7f2c780097b0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.622 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 shutdown_connections 2023-09-29T23:11:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:6808/113227,v1:172.21.15.121:6809/113227] conn(0x7f2c64098780 0x7f2c6409ac30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c801422c0 0x7f2c80068a90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c8006acb0 0x7f2c8009fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 --2- 172.21.15.121:0/3705020821 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c8006b080 0x7f2c80068550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-29T23:11:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 >> 172.21.15.121:0/3705020821 conn(0x7f2c8006d160 msgr2=0x7f2c80070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-29T23:11:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 shutdown_connections 2023-09-29T23:11:32.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-29T23:11:32.622+0000 7f2c8781e700 1 -- 172.21.15.121:0/3705020821 wait complete. 2023-09-29T23:11:32.633 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:11:32.636 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-29T23:11:32.636 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-09-29T23:11:32.636 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.644 INFO:tasks.ceph.mds.a:Stopped 2023-09-29T23:11:32.644 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-09-29T23:11:32.644 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.693 INFO:tasks.ceph.mds.c:Stopped 2023-09-29T23:11:32.693 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-09-29T23:11:32.693 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.703 INFO:tasks.ceph.mds.b:Stopped 2023-09-29T23:11:32.703 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-29T23:11:32.704 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-29T23:11:32.704 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.762 INFO:tasks.ceph.osd.0:Stopped 2023-09-29T23:11:32.762 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-29T23:11:32.762 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.792 INFO:tasks.ceph.osd.1:Stopped 2023-09-29T23:11:32.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-29T23:11:32.793 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.859 INFO:tasks.ceph.osd.2:Stopped 2023-09-29T23:11:32.859 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-29T23:11:32.860 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.924 INFO:tasks.ceph.osd.3:Stopped 2023-09-29T23:11:32.924 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-29T23:11:32.925 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-29T23:11:32.925 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.947 INFO:tasks.ceph.mgr.x:Stopped 2023-09-29T23:11:32.947 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-29T23:11:32.947 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:32.990 INFO:tasks.ceph.mgr.y:Stopped 2023-09-29T23:11:32.991 DEBUG:tasks.ceph.mgr.z:waiting for process to exit 2023-09-29T23:11:32.991 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:33.056 INFO:tasks.ceph.mgr.z:Stopped 2023-09-29T23:11:33.056 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-29T23:11:33.056 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-29T23:11:33.056 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:33.066 INFO:tasks.ceph.mon.a:Stopped 2023-09-29T23:11:33.066 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-29T23:11:33.066 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:33.116 INFO:tasks.ceph.mon.c:Stopped 2023-09-29T23:11:33.117 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-29T23:11:33.117 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:11:33.128 INFO:tasks.ceph.mon.b:Stopped 2023-09-29T23:11:33.128 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-29T23:11:33.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v 'replacing it with standby' | egrep -v 'No standby daemons available' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_HOST_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_FLAGS\)' | egrep -v '\(TELEMETRY_CHANGED\)' | egrep -v 'pauserd,pausewr flag\(s\) set' | egrep -v 'Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running' | egrep -v 'evicting unresponsive client .+' | egrep -v MON_DOWN | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-29T23:11:33.189 INFO:teuthology.orchestra.run.smithi121.stdout:1696028761.5075474 mgr.x (mgr.4107) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:11:33.189 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-29T23:11:33.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v 'replacing it with standby' | egrep -v 'No standby daemons available' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_HOST_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_FLAGS\)' | egrep -v '\(TELEMETRY_CHANGED\)' | egrep -v 'pauserd,pausewr flag\(s\) set' | egrep -v 'Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running' | egrep -v 'evicting unresponsive client .+' | egrep -v MON_DOWN | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-29T23:11:33.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v 'replacing it with standby' | egrep -v 'No standby daemons available' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_HOST_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_FLAGS\)' | egrep -v '\(TELEMETRY_CHANGED\)' | egrep -v 'pauserd,pausewr flag\(s\) set' | egrep -v 'Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running' | egrep -v 'evicting unresponsive client .+' | egrep -v MON_DOWN | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-29T23:11:33.335 INFO:teuthology.orchestra.run.smithi121.stdout:1696028761.5075474 mgr.x (mgr.4107) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-29T23:11:33.335 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-29T23:11:33.335 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-29T23:11:33.591 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:11:33.591 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-29T23:11:33.707 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:11:33.708 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-29T23:11:33.778 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:11:33.778 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-29T23:11:33.860 INFO:tasks.ceph:Archiving mon data... 2023-09-29T23:11:33.861 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/data/mon.a.tgz 2023-09-29T23:11:33.861 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-09-29T23:11:33.880 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.xbKjzKTyiJ 2023-09-29T23:11:33.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.xbKjzKTyiJ 2023-09-29T23:11:34.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.xbKjzKTyiJ 2023-09-29T23:11:34.098 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.xbKjzKTyiJ is 312KB 2023-09-29T23:11:34.194 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.xbKjzKTyiJ 2023-09-29T23:11:34.212 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/data/mon.c.tgz 2023-09-29T23:11:34.212 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-09-29T23:11:34.269 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.ptCxZdCv95 2023-09-29T23:11:34.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ptCxZdCv95 2023-09-29T23:11:34.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.ptCxZdCv95 2023-09-29T23:11:34.503 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.ptCxZdCv95 is 438KB 2023-09-29T23:11:34.599 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.ptCxZdCv95 2023-09-29T23:11:34.618 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/data/mon.b.tgz 2023-09-29T23:11:34.619 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-09-29T23:11:34.641 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.CDxM15A6rR 2023-09-29T23:11:34.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.CDxM15A6rR 2023-09-29T23:11:34.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.CDxM15A6rR 2023-09-29T23:11:34.893 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.CDxM15A6rR is 438KB 2023-09-29T23:11:34.988 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.CDxM15A6rR 2023-09-29T23:11:35.006 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-29T23:11:35.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:11:35.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:11:35.072 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-29T23:11:35.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-29T23:11:35.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-29T23:11:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-29T23:11:35.141 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-29T23:11:35.146 INFO:tasks.ceph:Archiving crash dumps... 2023-09-29T23:11:35.148 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/remote/smithi121/crash 2023-09-29T23:11:35.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-29T23:11:35.180 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/remote/smithi123/crash 2023-09-29T23:11:35.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-29T23:11:35.213 INFO:tasks.ceph:Compressing logs... 2023-09-29T23:11:35.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:11:35.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:11:35.964 INFO:tasks.ceph:Archiving logs... 2023-09-29T23:11:35.965 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/remote/smithi121/log 2023-09-29T23:11:35.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-29T23:11:36.183 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/remote/smithi123/log 2023-09-29T23:11:36.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-29T23:11:36.296 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:11:36.362 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c0dedf9e737141caa280e0336ea6f2e2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-29T23:11:36.365 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-29T23:11:36.385 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-29T23:11:36.393 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-29T23:11:36.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-29T23:11:36.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-29T23:11:36.485 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-29T23:11:36.494 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-29T23:11:36.495 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:11:36.497 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:11:36.512 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-09-29T23:11:36.514 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-09-29T23:11:36.516 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:11:36.516 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-09-29T23:11:36.517 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 -207us[ -190us] +/- 78ms 2023-09-29T23:11:36.517 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -438us[ -421us] +/- 62ms 2023-09-29T23:11:36.517 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +771us[ +788us] +/- 66ms 2023-09-29T23:11:36.517 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:11:36.519 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:11:36.519 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-09-29T23:11:36.519 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 -178us[ -178us] +/- 78ms 2023-09-29T23:11:36.519 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -446us[ -429us] +/- 62ms 2023-09-29T23:11:36.519 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +781us[ +798us] +/- 66ms 2023-09-29T23:11:36.519 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:11:36.520 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-29T23:11:36.533 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-29T23:11:36.533 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-29T23:11:36.542 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-29T23:11:36.649 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-29T23:11:36.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:11:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:11:36.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:11:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:11:36.802 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-29T23:11:36.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-29T23:11:36.820 INFO:teuthology.task.internal:Duration was 1041.786943 seconds 2023-09-29T23:11:36.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-29T23:11:36.829 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-29T23:11:36.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-29T23:11:36.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-29T23:11:36.877 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:11:36.889 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:11:37.178 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-29T23:11:37.178 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2023-09-29T23:11:37.178 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-29T23:11:37.201 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2023-09-29T23:11:37.201 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-29T23:11:37.224 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-29T23:11:37.225 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:11:37.244 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:11:37.303 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-29T23:11:37.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-29T23:11:37.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-29T23:11:37.409 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-29T23:11:37.419 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-29T23:11:37.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-29T23:11:37.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-29T23:11:37.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-29T23:11:37.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:11:37.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:11:37.514 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2023-09-29T23:11:37.542 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2023-09-29T23:11:37.571 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:11:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:11:37.604 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:11:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:11:37.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-29T23:11:37.638 INFO:teuthology.task.internal:Transferring archived files... 2023-09-29T23:11:37.638 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/remote/smithi121 2023-09-29T23:11:37.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-29T23:11:37.682 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517/remote/smithi123 2023-09-29T23:11:37.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-29T23:11:37.719 INFO:teuthology.task.internal:Removing archive directory... 2023-09-29T23:11:37.720 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-29T23:11:37.722 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-29T23:11:37.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-29T23:11:37.783 INFO:teuthology.task.internal:Not uploading archives. 2023-09-29T23:11:37.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-29T23:11:37.853 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-29T23:11:37.854 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-29T23:11:37.856 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-29T23:11:37.871 INFO:teuthology.orchestra.run.smithi121.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 29 23:11 /home/ubuntu/cephtest 2023-09-29T23:11:37.873 INFO:teuthology.orchestra.run.smithi123.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 29 23:11 /home/ubuntu/cephtest 2023-09-29T23:11:37.875 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-29T23:11:37.884 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-29T23:11:37.903 INFO:teuthology.nuke:Checking targets against current locks 2023-09-29T23:11:37.930 DEBUG:teuthology.nuke:shortname: smithi121 2023-09-29T23:11:37.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:11:37.954 DEBUG:teuthology.nuke:shortname: smithi123 2023-09-29T23:11:37.954 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:11:37.958 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:45:37.690018', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4/SG/jST+P3fpYnvP3qqBOCdxntMhW67oJi2kYDwucLwJ6BPd99WBtmrWt5kYaiaQ0UR/qpZ7k/fy4wK9GW/g='} 2023-09-29T23:11:37.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406517', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 22:45:37.691951', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArobUAH9Zdgerb63Cfvmh72MtB0kienk5a9ay4DNdK59tgLV3YNl1ZfokzHBBrrcpYP12D7NyCVsOU0Gco9sTA='} 2023-09-29T23:11:38.008 INFO:teuthology.orchestra.console.smithi121:Power off 2023-09-29T23:11:38.008 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-29T23:11:38.032 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-09-29T23:11:38.032 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:11:38.048 INFO:teuthology.orchestra.console.smithi123:Power off 2023-09-29T23:11:38.048 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-29T23:11:38.057 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-09-29T23:11:38.068 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2023-09-29T23:11:38.068 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:11:38.089 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-09-29T23:11:42.058 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:11:42.090 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:11:42.168 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-09-29T23:11:42.199 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-09-29T23:11:46.174 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:11:46.201 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:11:46.283 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-09-29T23:11:46.309 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2023-09-29T23:11:46.385 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-09-29T23:11:46.410 INFO:teuthology.orchestra.console.smithi123:Power off completed 2023-09-29T23:11:46.515 INFO:teuthology.run:Summary data: description: rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/dashboard} duration: 1041.7869431972504 failure_reason: '"1696028761.5075474 mgr.x (mgr.4107) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c0dedf9e737141caa280e0336ea6f2e2 status: fail success: false 2023-09-29T23:11:46.515 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T23:11:46.594 INFO:teuthology.run:FAIL